Refrigerator with camera and control method for the same

ABSTRACT

A refrigerator includes a storage compartment formed in a cabinet of the refrigerator by a fixed insulation wall, the storage compartment being provided with an access opening. The refrigerator also includes a door rotatably provided to the cabinet to open and close the access opening, a drawer provided in the storage compartment, a camera fixed to a ceiling of the storage compartment to photograph both a first region for storing of food arranged in an external space of the drawer and a second region for storing of food arranged in an internal space of the drawer, and a controller configured to separate, through a capturing time of a picture containing both the first region and the second region, a first region picture and a second region picture from the picture to individually divide and store the first region picture and the second region picture.

CROSS-REFERENCE TO RELATED APPLICATION

Pursuant to 35 U.S.C. § 119(a), this application is a continuation ofU.S. application Ser. No. 14/930,012, filed Nov. 2, 2015, which is acontinuation of U.S. application Ser. No. 14/786,719, filed Oct. 23,2015, now published, which claims the benefit of an earlier filing dateand right of priority to International Application No. PCT/KR2014/003523filed on Apr. 22, 2014, which claims the benefit of foreign priorityapplication(s) filed in Korea as Serial No. 10-2013-0044528 on Apr. 23,2013, Serial No. 10-2013-0124739 on Oct. 18, 2013, Serial Nos.10-2014-0045066, 10-2014-0045067 and 10-2014-0045068 on Apr. 15, 2014,the contents of which are incorporated by reference herein in theirentirety.

TECHNICAL FIELD

The present invention relates to a refrigerator and a control method forthe same, and, more particularly, to a refrigerator capable of providinginformation about food stored in the refrigerator even without the doorof the refrigerator being opened by a user and a control method for thesame.

BACKGROUND

In general, a refrigerator, which is an apparatus for supplying cooledair according to driving of a refrigeration cycle, serves to store foodat a low temperature. Conventional refrigerators have only functioned tostore food at a low temperature. Recently, however, additional functionsother than the function of storing food have been increasingly needed.To check the interior of a refrigerator serving to receive and storecertain items, the door of the refrigerator need to be opened. Inaddition, in the case that the quantity and kinds of food stored in therefrigerator are not identified when a user desires to buy an item in amarket or a mart, a food item may be redundantly purchased, or anecessary food item may not be purchased.

In conventional art, Japanese Patent No. 3450907 and Japanese PatentApplication Publication No. 2004-183987 disclose that a camera isinstalled at a door to photograph the interior of the refrigerator. Inaddition, Japanese Patent Application Publication No. 2001-294308discloses that cameras are installed in the refrigerator, in a drawer,and at the door.

However, according to the aforementioned conventional documents, therange of photographing of the camera installed in the refrigerator islimited. Thereby, a plurality of cameras needs to be used to photographvarious storage regions.

Accordingly, it is needed to improve the photographing efficiency of acamera by reducing the number of cameras installed in the refrigeratorand increasing the range of a region photographed by the camera.

In addition, according to conventional art, there have been variousproblems in acquiring a photo of the interior of the drawer. Forexample, a specific method of acquiring a picture captured at a desiredtime, concern about dew condensation occurring on the camera, andimprovement of power consumption of the camera have not beenspecifically addressed.

More specifically, the conventional art have been lacked concreteness ofproviding optimum information for the user in consideration ofuniqueness and positional relationship of a plurality of storage regionsin the case the storage regions including a drawer are present in therefrigerator.

SUMMARY

The present invention is basically devised to solve the aforementionedproblem.

An object of the present invention devised to solve the problem lies ona storage compartment display device of a refrigerator configured toprovide information about food stored in the refrigerator without thedoor of the refrigerator being opened by a user and a control method forthe same.

Another object of the present invention devised to solve the problemlies on a refrigerator allowing the user to intuitively recognizeinformation about food stored in a plurality of storage regions in astorage compartment and a control method for the same.

Another object of the present invention devised to solve the problemlies on a refrigerator allowing the user to intuitively recognize thepositional relationship between plural storage regions and capable ofproviding information about food stored to substantially coincide withthe user's point of view and a control method for the same.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of operatively connecting the time todrive a camera the time to perform photographing operation with openingof the door and/or a rotation angle of the door to reduce powerconsumption of the camera and acquire information about the stored foodthrough the camera at the optimum time, and a control method for thesame.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of preventing a picture quality frombeing degraded due to dew condensation.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of providing the user with the latestinformation about food stored in the refrigerator and a control methodfor the same.

Another object of the present invention devised to solve the problemlies on a refrigerator allowing a manager or user to correct regions ofa picture displayed for the user to provide the user with theinformation about stored food in an optimum state and a control methodfor the same.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of provide the user with optimuminformation about stored food by fixing a camera to a cabinet to preventthe camera from being shaken and a control method for the same.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of simplifying the configuration byidentifying information about the state of a drawer through software andeffectively identifying and providing information about food stored inthe drawer and a control method for the same.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of providing the latest information aboutstored food in each of the plural storage regions including a drawer bycontinuously performing the photographing operation through one cameraand a control method for the same.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of providing, using one camera,information about food stored in the inner space of the drawer andanother storage region at least partially overlapping the inner space ofthe drawer in a vertical direction or in the inner space of the drawerand the inner space of another drawer at least partially overlapping theinner space of the drawer in a vertical direction and a control methodfor the same.

Another object of the present invention devised to solve the problemlies on a refrigerator and a control method for the same which mayprevent the quality of pictures captured through a camera from beingdegraded due to dew condensation.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of reducing necessary expenses for therefrigerator by limiting the number of cameras installed in therefrigerator to one.

Another object of the present invention devised to solve the problemlies on a refrigerator capable of effectively prevent power consumptionfrom increasing due to the camera and a control method for the same.

A further object of the present invention devised to solve the problemlies on a refrigerator capable of minimizing load to a controller and amemory unit and effectively processing continuously captured pictures.

Solution to Problem

The object of the present invention can be achieved by providing arefrigerator including a storage compartment formed in a cabinet of therefrigerator by a fixed insulation wall, the storage compartment beingprovided with an access opening, a door rotatably provided to thecabinet to open and close the access opening, a drawer provided in thestorage compartment, a camera fixed to a ceiling of the storagecompartment to photograph both a region (a first region) for storing offood arranged in an external space of the drawer in the storagecompartment and a region (a second region) for storing of food arrangedin an internal space of the drawer in the storage compartment, and acontroller to separate, through a capturing time of a picture containingboth the first region and the second region, a portion for the firstregion (a first region picture) and a portion for the second region (asecond region picture) from the picture to individually divide and storethe first region picture and the second region picture.

In another aspect of the present invention, provided herein is arefrigerator including a storage compartment formed in a cabinet of therefrigerator by a fixed insulation wall, the storage compartment beingprovided with an access opening, a door rotatably provided to thecabinet to open and close the access opening, at least one drawerprovided in the storage compartment, a camera fixed to a ceiling of thestorage compartment to photograph both a region (a first region) forstoring of food arranged in an external space of the drawer in thestorage compartment and a region (a second region) for storing of foodarranged in an internal space of the drawer in the storage compartment,a controller to separate, through a capturing time of a picturecontaining both the first region and the second region, a portion forthe first region (a first region picture) and a portion for the secondregion (a second region picture) from the picture to individually divideand store the first region picture and the second region picture, and adisplay to display the first region picture and the second regionpicture such that the first region picture and the second region aredivided from each other.

In another aspect of the present invention, provided herein is arefrigerator including a storage compartment formed in a cabinet of therefrigerator by a fixed insulation wall, the storage compartment beingprovided with an access opening, a door rotatably provided to thecabinet to open and close the access opening, at least one drawerprovided in the storage compartment, a camera fixed to a ceiling of thestorage compartment to photograph both a region (a first region) forstoring of food arranged in an external space of the drawer in thestorage compartment and a region (a second region) for storing of foodarranged in an internal space of the drawer in the storage compartment,and a controller to separate a portion for the first region (a firstregion picture) from a picture taken at a certain time when the door isclosed after being opened and separates a portion for the second region(a second region picture) from a picture taken at a certain time whenthe drawer is introduced after being withdrawn to individually separate,divide, store and update the first region picture and the second regionpicture.

In another aspect of the present invention, provided herein is arefrigerator including a storage compartment, a drawer movably providedin the storage compartment, the drawer including with a marker, a camerafixedly provided to a ceiling of the storage compartment to photographthe drawer from an outside of the drawer, and a controller to sense aposition of the marker in pictures continuously taken through the camerato determine state information about the drawer including at least of adegree of withdrawal of the drawer, whether the drawer is withdrawn, amovement direction of the drawer, and a stopped state or moving state ofthe drawer.

In another aspect of the present invention, provided herein is arefrigerator including a storage compartment formed in a cabinet of therefrigerator by a fixed insulation wall, the storage compartment beingprovided with an access opening, a door rotatably provided to thecabinet to open and close the access opening, at least one drawerprovided in the storage compartment, at least one shelf provided in thestorage compartment, the at least one shelf being positioned at an upperside of the drawer, and a camera fixed to a ceiling of the storagecompartment between a front edge of the shelf and the access opening tophotograph both a region (a first region) provided in an external upperspace to allow the shelf to store food and a region (a second region)provided in an internal space of the drawer to store.

In another aspect of the present invention, provided herein is arefrigerator including a storage compartment configured with an accessopening in a front surface thereof and an upper wall, a lower wall, bothsidewalls and a rear wall and provided therein with a shelf regiondivided by a plurality of shelves and a drawer region having at leastone drawer, the upper wall, lower wall, sidewalls and rear wall beingformed of an insulating material, a cabinet provided therein with thestorage compartment, at least one door closely contacting the frontsurface of the storage compartment to open and close the storagecompartment, and a camera device installed at the upper wall of thestorage compartment between the access opening of the storagecompartment and a front edge of the shelves installed in therefrigerator, wherein the camera device includes a camera moduleconfigured to receive a camera lens and electrical components needed todrive a camera and provided with a transparent window spaced a certaindistance from the camera lens, and a camera housing part configured toseat and fix the camera module part at a determined position in thecamera housing, wherein the camera housing includes a fixing surfaceadjoining the upper wall of the storage compartment, a front surfaceformed to face the access opening of the storage compartment, both sidesurfaces connecting the front surface to a rear surface, and a topsurface provided with an opening allowing a camera window of the cameramodule to be exposed therethrough, wherein the front surface of thecamera housing is formed at a lower position than the opening formed inthe top surface of the camera housing.

In another aspect of the present invention, provided herein is Arefrigerator including one camera to take a picture of an interior of astorage compartment for storing food, a controller to divide one picturetaken by the camera into a plurality of images to manage the picture,and a display to independently display the images, wherein the storagecompartment includes a first region having food placed on a shelf, asecond region defined by an internal space of a drawer introduced intoor withdrawn from a lower space of the shelf, and a third region formedon a bottom of a front of the drawer, the third region overlapping thesecond region when the drawer is withdrawn from the lower space, whereinthe one camera photographs the first region, the second region and thethird region.

In another aspect of the present invention, provided herein is a controlmethod for a refrigerator including sensing whether a door rotates anangle greater than or equal to a certain angle to open a storagecompartment, taking, with one camera, one picture including a firstregion having food placed on a shelf, a second region defined by aninternal space of a drawer introduced into or withdrawn from a lowerspace of the shelf and a third region formed on a bottom of a front ofthe drawer to overlap the second region when the drawer is withdrawnfrom the lower space, dividing the one picture into a first image of thefirst region, a second image of the second region, and a third image ofthe third region, individually updating at least one of the first image,second image and third image, and displaying the updated image.

In another aspect of the present invention, provided herein is a controlmethod for a refrigerator provided with a drawer and a camera includingrecognizing closing of the drawer, acquiring a final image of aninterior of the drawer using the camera at a moment the closing thedrawer begins, and displaying the final image.

In another aspect of the present invention, provided herein is arefrigerator including a withdrawable drawer, a drawer sensing unit tosense closing or opening of the drawer, a camera to acquire an image ofan interior of the drawer, and a controller to control, when the closingof the drawer is recognized, the camera to acquire a final image of theinterior of the drawer at the drawer at a moment the closing of thedrawer begins.

Advantageous Effects of Invention

According to embodiments of the present invention, a user does not needto open the door of a refrigerator to obtain information about foodstored in the refrigerator. Thereby, leakage of cooled air from thestorage compartment may be prevented. Accordingly, unnecessary loss ofcooled air may be prevented and the energy efficiency of therefrigerator may be improved.

In addition, according to embodiments of the present invention, thelatest information about the food stored in the refrigerator may beprovided for the user. Therefore, reliability of the information aboutthe stored food provided for the user may be enhanced.

Further, according to embodiments of the present invention, informationabout food items stored in various positions may be provided by a singlecamera. Accordingly, a structure allowing installation of only onecamera is added, and thus designing of the refrigerator may befacilitated. Particularly, costs incurred by use of a camera may bereduced.

According to embodiments of the present invention, dew condensation on acamera installed in the refrigerator may be prevented. Accordingly, animage captured by the camera may be stably provided for the user.

According to embodiments of the present invention, a user may remotelyfigure out the current internal situation of the refrigerator andreceive necessary information about food from an outside provider.

According to embodiments of the present invention, the refrigerator mayprovide a screen of a photo of the interior of a drawer captured by acamera which is similar to what the user actually sees when the useruses the refrigerator.

According to embodiments of the present invention, the refrigerator mayprovide the user with planar images of spatially overlapping andinvisible locations on one screen.

All or part of the features described throughout this application can beimplemented as a computer program product including instructions thatare stored on one or more non-transitory machine-readable storage media,and that are executable on one or more processing devices. All or partof the features described throughout this application can be implementedas an apparatus, method, or electronic system that can include one ormore processing devices and memory to store executable instructions toimplement the stated functions.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims. Thedescription and specific examples below are given by way of illustrationonly, and various changes and modifications will be apparent.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention, illustrate embodiments of the inventionand together with the description serve to explain the principle of theinvention.

In the drawings:

FIG. 1 is a front view illustrating a refrigerator according to onepossible implementation;

FIG. 2 is a view illustrating a door of FIG. 1, which is in an openedposition;

FIG. 3 is a view illustrating a third region disposed at the bottom ofthe storage compartment;

FIGS. 4A and 4B are control block diagrams illustrating oneimplementation;

FIG. 5 is a view illustrating an angle of view of the camera;

FIG. 6 is a transverse cross-sectional view of FIG. 5;

FIG. 7 shows an image captured by the camera with the constituentspositioned as shown in FIG. 6

FIG. 8 is a view illustrating selection of a position of the camera;

FIG. 9 is a cross-sectional view showing main parts of the refrigerator.

FIG. 10 is a view illustrating operation of the door sensors;

FIG. 11 is a view specifically illustrating a left hinge unit and righthinge unit;

FIGS. 12A to 12C show screens provided to the user;

FIG. 13 is a view illustrating adjustment of a photo captured by thecamera;

FIG. 14 is a perspective view showing the camera;

FIG. 15 is a view illustrating main parts of the camera;

FIG. 16 is a cross-sectional view of FIG. 14;

FIG. 17 is a view illustrating disposition of the heater;

FIGS. 18A to 18C are views showing photos captured with the cameramounted to the refrigerator such that the camera is vertically inclined;

FIGS. 19A to 19C are views showing photos captured with the cameramounted to the refrigerator such that the camera is horizontallyinclined;

FIG. 20 is a view illustrating a camera housing in an assembled state;

FIG. 21 is a front view showing a first housing;

FIG. 22 is a front view showing a second housing;

FIG. 23 is a front view showing the first housing with the camerainstalled;

FIG. 24 is a transverse cross-sectional view of FIG. 23;

FIG. 25 is a cross-sectional view illustrating the cameral housinginstalled at the inner case;

FIG. 26 shows a table comparing power consumption of a camera accordingto one implementation with a counterpart example;

FIGS. 27A and 27B are views comparing the supplied standby current withthe drive current;

FIGS. 28 and 29 are views illustrating the start time of photographingby the camera and continuous photographing by the camera;

FIG. 30 is a view illustrating a drawer sensor according to oneimplementation;

FIGS. 31A and 31B are views illustrating a method for the drawer sensorof FIG. 30 to sense movement of the drawer;

FIG. 32 is a view showing a marker indicated on the drawer;

FIG. 33A is a view illustrating a photo showing the left and rightdrawers which are in the withdrawn position;

FIG. 33B is a view illustrating a photo showing the left drawer which isin the introduced position and the right drawer which is in thewithdrawn position.

FIG. 34 is a flowchart describing operation of the refrigeratoraccording to one implementation;

FIG. 35 is a flowchart describing a method of recognizing and tracingthe marker according to one implementation;

FIGS. 36A and 36B are views illustrating a marker for sensing the timewhen the drawer is introduced or withdrawn according to oneimplementation;

FIGS. 37A and 37B are views illustrating the shape of a marker accordingto one implementation of the present invention;

FIG. 38 is a flowchart illustrating a method of sensing closing of thedrawer according to one implementation;

FIG. 39 is a flowchart illustrating an operation of the refrigeratorthat is performed when termination of closing of the drawer is sensedaccording to one implementation;

FIG. 40 is a flowchart illustrating a method of controlling an image ofa certain region in the refrigerator captured at a certain timeaccording to another implementation;

FIGS. 41A and 41B are views illustrating various forms of the marker;

FIG. 42 is a view illustrating a method of recognizing the position ofthe marker;

FIGS. 43A to 43C are views illustrating the degree of access opening ofthe drawer.

FIG. 44 is a view illustrating movement of the marker;

FIG. 45 is a control flowchart according to one implementation;

FIG. 46 is a control flowchart illustrating a variation of theimplementation illustrated in FIG. 45;

FIG. 47 is a control flowchart illustrating a variation of theimplementation of FIG. 46;

FIG. 48 is a control flowchart illustrating another variation of theimplementation of FIG. 45;

FIG. 49 is a view illustrating another variation of the implementationof FIG. 45;

FIG. 50 is a control flowchart illustrating another implementation;

FIG. 51 is a view illustrating a process of updating images with twodoors for access opening and closing the storage compartment and twodrawers provided.

FIG. 52 is a ladder diagram illustrating a method of operating therefrigerator according to another implementation;

FIG. 53 is a ladder diagram illustrating a method of operating therefrigerator according to another implementation;

FIG. 54 is a ladder diagram illustrating a method of operating therefrigerator according to another implementation;

FIG. 55 is a ladder diagram illustrating a method of operating therefrigerator according to another implementation;

FIGS. 56A and 56B are views illustrating operation of a heater of thecamera;

FIGS. 57A and 57B are views illustrating a result of an experiment ondew condensation occurring on a transparent window of the cameraaccording to temperatures;

FIG. 58 is a cross-sectional view illustrating a transparent window;

FIGS. 59A to 60B are views schematically illustrating installation ofthe camera at the inner case;

FIG. 61 is a view illustrating a refrigerator according to anotherimplementation;

FIG. 62 is a view illustrating a screen provided for the user from therefrigerator of FIG. 61; and

FIG. 63 is a view illustrating a method of adjusting a picture capturedwith a camera of the refrigerator of FIG. 61.

DETAILED DESCRIPTION Best Mode for Carrying Out the Invention

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings.

FIG. 1 is a front view illustrating a refrigerator according toembodiments of the present invention, and FIG. 2 is a view illustratinga door of FIG. 1, which is in an opened position. Hereinafter, adescription will be given with reference to FIGS. 1 and 2.

The refrigerator according to embodiments is applicable to a topmount-type refrigerator having a fresh food compartment and freezercompartment horizontally divided with the freezer compartment disposedon the fresh food compartment and a side by side-type refrigeratorhaving the freezer compartment and the fresh food compartment which arevertically divided.

In the embodiments disclosed below, descriptions will be given of abottom freezertype refrigerator having a freezer compartment and freshfood compartment which are horizontally divided with the freezercompartment disposed under the fresh food compartment.

The body of the refrigerator includes an outer case 10 forming anoverall exterior of the refrigerator seen to the user and an inner case12 forming a storage compartment 22 to store food. A predetermined spacemay be formed between the outer case 10 and the inner case 12 to form apassage allowing cooled air to circulate therethrough. In addition, aninsulation material may fill the space between the outer case 10 and theinner case 12 to maintain the interior of the storage compartment 22 ata low temperature relative to the exterior of the storage compartment22.

In addition, a refrigeration cycle system configured to circulate arefrigerant to produce cooled air is installed in a machine chamber (notshown) formed in the space between the outer case 10 and the inner case12. The refrigeration cycle system may be used to maintain the interiorof the refrigerator at a low temperature to keep freshness of the foodstored in the refrigerator. The refrigeration cycle system includes acompressor configured to compress the refrigerant and an evaporator (notshown) configured to change the refrigerant from the liquid state to thegaseous state such the refrigerant exchanges heat with the exterior.

The refrigerator is provided with doors to open the storage compartment.Herein, the doors may include a freezer compartment door 30 and a freshfood storage compartment door 20. One end of each of the doors ispivotably installed at the body of the refrigerator. A plurality offreezer compartment doors 30 and a plurality of fresh food storagecompartment doors 20 may be provided. That is, as shown in FIG. 2, thefreezer compartment doors 30 and fresh food storage compartment doors 20may be installed to be opened forward by rotating about both edges ofthe refrigerator.

The space between the outer case 10 and the inner case 12 may be filledwith a foaming agent to insulate the storage compartment 22.

An insulated space is formed in the storage compartment 22 by the innercase 12 and the door 20. Once the storage compartment 22 is closed bythe door 20, an isolated and insulated space may be formed therein. Inother words, the storage compartment 22 is isolated from the externalenvironment by the insulation wall of the door 200 and the insulationwall of the cases 10 and 12.

Cooled air supplied from the machine chamber may flow everywhere in thestorage compartment 22. Accordingly, the food stored in the storagecompartment 22 may be maintained at a low temperature.

The inner case 12 may be provided with a barrier 60 formed at the bottomof the storage compartment 22. The barrier 60 may be installed at thelower end of the storage compartment 22 to divide the storagecompartment into the freezer compartment and the fresh food compartment.The barrier 60 may have a predetermined thickness and be formed at theinner case 12. The barrier 60 may extend horizontally.

The storage compartment 22 may include a shelf 40 on which food itemsare placed. Herein, the storage compartment 22 may include a pluralityof shelves 40, and food items may be placed on each of the shelves 40.The shelves 40 may be positioned horizontally to partition the interiorof the storage compartment.

A drawer 50 is installed in the storage compartment 22 such that thedrawer 50 may be introduced into or withdrawn from the storagecompartment 22. Items, for example, food items are accommodated andstored in the drawer 50. It may be possible to dispose two drawers 50side by side in the storage compartment 22. The user may open the leftdoor of the storage compartment 22 to reach the drawer disposed on theleft side. On the other hand, the user may open the right door of thestorage compartment 22 to reach the drawer disposed on the right side.

A space to store food may be formed in the barrier 60. Since the barrier60 is provided on the inner case 12 rather than on the doors, it is heldstationary when the doors rotates. This allows the user to stably storeor retrieve food.

The interior of the storage compartment 22 may be partitioned into aspace positioned over the shelves 40, a space formed by the drawer 50,and a space formed by the barrier 60. Thereby, a plurality ofpartitioned spaces to store food may be provided.

Herein, cooled air provided to the storage compartment 22 may move toeach of the spaces arranged in one storage compartment 22. That is, thepartitioned spaces allow the cooled air to move thereinto, and there thespaces have a different meaning than the aforementioned storagecompartment.

Specifically, there may be temperature difference between the spaces,but the spaces do not block transfer of heat therebetween, unlike thestorage compartment which forms the insulated space.

The cooled air supplied to one storage compartment may not be allowed tofreely move to the storage compartment, but it may be allowed to freelymove to the partitioned spaces installed in one storage compartment.That is, the cooled air positioned over the shelf 40 is allowed to moveinto the space formed by the drawer 50.

The refrigerator according to some implementations may also include acamera 70 configured to capture images of the interior of the storagecompartment 22. Herein, the camera 70 may be maintained at a fixedposition to capture images of the same part.

Specifically, the camera 70 may be installed at the upper wall of theinner case 12 to face downward. Thereby, it may capture images of fooditems stored in the storage compartment 22. A captured image may have aview of food seen to the user when the user looks down the interior,i.e., the view that the user will see when the user actually uses therefrigerator.

Specifically, the camera 70 may be installed at a position correspondingto the interior of the drawer which is fully open. Thereby, a capturedimage may provide a figure similar to the view seen by the user when theuser looks down the interior of the drawer.

Herein, the storage compartment 22 may include a first region 42 inwhich food is placed on the shelf 40 and a second region 52 defined asthe inner space of the drawer 50. The storage compartment 22 may alsoinclude a third region 62 other than the first region 42 and the secondregion 52. The third region 62 may overlap the second region 52 when thedrawer 50 is withdrawn, which will be described in more detail later.The first region 42 and the second region 52 may have different heightsin the storage compartment 22. That is, the height of first region 42may be greater than that of the second region 52.

Images captured by the camera 70 and various kinds of information aboutthe refrigerator may be provided for the user through a display 14arranged on the front surface of the refrigerator. In addition, the usermay control the refrigerator through the display 14.

The cooled air supplied to the storage compartment 22 may move into thefirst region 42 and the second region 52. The cooled air positioned inone region may move into the other region.

FIG. 3 is a view illustrating the third region disposed at the bottom ofthe storage compartment. Hereinafter, a description will be given withreference to FIG. 3.

The barrier 60 may be provided with a recessed portion, in which foodmay be stored. This portion may be defined as the third region 62. Thethird region 62 is the space formed in the barrier 60.

The third region, a storage space disposed at a position different fromthe positions of the first region and the second region, may provide theuser with another space for food storage.

Food items such as eggs which are frequently used may be stored in thethird region 62. To this end, a cover 68 to open the third region 62 maybe provided. In this case, the cover 68 is preferably formed of atransparent material to allow the camera 70 positioned at the upper sideto capture an image of the food stored in the third region 62 in spiteof the cover 68 being positioned between the camera 70 and the food.

The drawer 50 is disposed at the upper side of the barrier 60.Accordingly, with the drawer 50 positioned in the space under the shelf,the user can reach the third region 62.

On the other hand, with the drawer 50 withdrawn from the space under theshelf, the third region 62 is disposed at the upper side of the drawer50, and thus the user cannot reach the third region 62.

That is, with the drawer 50 withdrawn toward the user, the drawer 50 andthe third region 62 overlap each other. When the user looks down at thedrawer 50 in this situation, the interior of the drawer 50 may be seento the user, and the third region 62 may not be seen.

On the other hand, when the drawer 50 is positioned not to be withdrawntoward the user, the drawer 50 and the third region 62 are disposed notto overlap each other. When the user looks down at the third region 62,only the third region 62 may be seen and the interior of the drawer 50may not be visible to the user.

The first region 42, the second region 52 and the third region 62 mayhave different heights in the storage compartment 22. That is, theheight of the first region 42 may be greater than the second region 52and the third region 62, and the second region 52 may have a greaterheight than the third region 62.

The cooled air supplied to the storage compartment 22 may move to thefirst region 42, the second region 52, and the third region 62. Thecooled air positioned in each region may move to another region.

That is, an area whose image is captured by the camera 70 may be aplurality of storage spaces in the storage compartment 22. The camera 70captures images of the interior of one storage compartment 22, but oneimage may contain information about a plurality of partitioned storagespaces.

The camera 70 may capture a photo such that plural storage spaces areexposed on the photo. Particularly, it may be possible that differentstorage spaces are photographed depending upon the capture time.

For example, the camera disposed in a fixed position may selectivelyphotograph the first region and second region or selectively photographthe second region and the third region according to the positionalrelationship of the door or the drawer and the time at which a pictureis taken. A detailed description will be given later.

FIG. 4A is a control block diagram illustrating one implementation.Hereinafter, a description will be given with reference to FIG. 4A.

The illustrated example may include a controller 100 which divides aphoto captured by the camera 70 into a plurality of images to manage thephoto. The plurality of images may represent portions of the capturedphoto processed or corrected by the controller 100.

The controller 100 may command the camera 70 to capture a photo andreceive the captured photo transmitted from the camera 70.

In addition, the controller 100 may provide some of the divided imagesto the display 14 to provide the user with the latest information aboutthe food stored in the storage compartment 22. The display 14 may beinstalled on the front surface of the refrigerator, or may construct adevice separated from the refrigerator. That is, the user may receive animage related to the storage compartment 22 through an externalcommunication terminal such as a cell phone and acquire information.

The controller 100 may divide a photo captured by the camera 70 into aplurality of areas of independent images and provide the same to thedisplay 14. At this time, the photo selected by the controller 100 mayinclude information about the latest status of the storage compartmentafter the user most recently accessed the storage compartment 22 toretrieve or place food from or in the storage compartment 22.

According to this example, the refrigerator may include a 110 capable ofsensing whether the door 20 opens or closes the storage compartment 22.Herein, the door switch 110 may be provided to the outer case 10.Thereby, it may sense the door 20 closing the storage compartment 22when the door 20 contacts the outer case 10. In addition, the door 20does not contact the outer case 10, the door 20 may sense the door 20opening the storage compartment 22.

According to this embodiment, the refrigerator may also include a doorsensor 120 to sense an angle of rotation of the door 20. Herein, thedoor sensor 120 may sense a rotation direction and rotation angle of thedoor 20. For example, when the door 20 rotates by an angle greater thanor equal to a certain value, a change occurs in the door sensor 120.Thereby, the door sensor 120 may sense that the door 20 has rotated anangle greater than or equal to a certain value. In addition, when thedoor 20 rotates in a specific direction, the door sensor 120 may sensechange in pulses generated according to the rotation direction to sensethe rotation direction of the door 20. The door sensor 120 may bedifferently configured than the configuration described above.

The door sensor 120 may also be provided with a light emitting part anda light receiving part. Thereby, the door sensor 120 may determinewhether the light radiated from the light emitting part is transferredto the light receiving part to sense the rotation angle of the door 20.

Particularly, the door sensor 120 may be driven only when the doorswitch 110 determines that the door 20 has opened the storagecompartment 22.

Once the door sensor 120 determines that the door 20 has rotated aspecific angle, the camera 70 may capture a photo. In this case; thecamera 70 does not immediately start to capture a photo at the time thedoor switch 110 senses opening of the door 20, but captures a photo onlywhen the door sensor 120 reaches the specific angle.

Once the camera 70 stars capturing operation, it may keep capturingphotos at predetermined time intervals. When the door sensor 120determines that the door 20 is opened by a certain angle, the camera 70may continuously capture photos until a command to stop capturing photosis provided, rather than capturing only one photo.

The controller 100 may include a drawer sensing unit 130 to sensewithdrawal or introduction of the drawer 50. Herein, the drawer sensingunit 130 may represent a constituent in the controller 100 whichinterprets pictures taken by the camera 70. The drawer sensing unit 130may not only sense movement of the drawer 50, but also the direction ofmovement of the drawer 50.

Specifically, the drawer sensing unit 130 may be implemented bysoftware. The drawer sensing unit 130 may use captured information in aphoto captured by the camera 70 to sense the position of the drawer 50.

The drawer 50 may be provided with an indication called a marker toprovide information about whether the drawer 50 has entered a certainsection through the taken pictures of the marker, whether the drawer 50has stopped if the drawer 50 has entered the certain section, andwhether the direction of movement of the drawer 50 has changed afterstopping. The marker will be more specifically described later.

Since the marker is placed on the drawer, it makes the same movement asthe drawer 50. Accordingly, without providing a separate sensor to sensemovement of the drawer, various kind of information about movement ofthe drawer 50 may be acquired using information about the photo capturedby the camera 70.

The photo captured by the camera 70 may be stored in the storage unit18. Herein, the storage unit 18 may be provided to the refrigerator orto a device provided separately from the refrigerator. The storage unit18 and the display 14 may be disposed together at the refrigerator or atan apparatus different from the refrigerator, a server connected to therefrigerator and a network, or a terminal connected to the refrigeratorand a network.

Not all the photos captured by the camera 70 may be stored in thestorage unit 18.

For example, in the case that the controller 100 does not issue acommand to select a specific photo, an old photo stored in the storageunit 18 may be deleted and a currently captured photo may be stored inthe storage unit 18 (First In, First Out). The storage unit 18 may storeonly a part of the photos captured by the camera 70 to reduce thestorage amount.

FIG. 4B is a control block diagram illustrating a variation of theillustrated implementation. Hereinafter, a description will be givenwith reference to FIG. 4B.

In FIG. 4B illustrating an example different from the exampleillustrated in FIG. 4A, the drawer sensing unit 130 may sense movementof the drawer 50 without depending on the information acquired from aphoto captured by the camera 70.

For example, the drawer sensing unit 130 may be provided with aplurality of Hall sensors. The Hall sensors may be installed on a pathof movement of the drawer 50. Thereby, when the drawer 50 moves, changesin the respective Hall sensors may be sensed to determine the positionand movement direction of the drawer 50.

By sensing movement of the drawer 50, the drawer sensing unit 130 mayestimate the time at which the internal view of the drawer 50 appears ina photo captured by the camera 70 while the photo capture is conductedby the camera 70. That is, the drawer sensing unit 130 may sense thetime at which the information about the items stored in the drawer 50 ispresented in a photo captured by the camera 70.

By sensing movement of the drawer 50, the drawer sensing unit 130 maysense the time the latest status of the drawer 50 is determined afterthe drawer 50 is used by the user. That is, the drawer sensing unit 130may sense the time at which the latest information is included in aphoto captured by the camera 70 after the user finishes using thedrawer. That is, the time at which introduction or retrieval of an itemis terminated or updating the items is terminated may be indirectlyidentified.

FIG. 5 is a view illustrating an angle of view of the camera.Hereinafter, a description will be given with reference to FIG. 5.

In FIG. 5, the drawer 50 is withdrawn and thus the camera 70 is allowedto photographs the second region 52. The shelf is omitted for simplicityof illustration.

The camera 70 may be disposed at the upper wall of the inner case 12such that it is inclined toward the rear wall of the inner case 12.

A photo captured by the camera 70 may have a horizontal angle of view inthe direction of width of the storage compartment 22 and a verticalangle of view in the front-to-back direction of the storage compartment22.

Preferably, the horizontal angle according to the camera 70 is set suchthat the inner space of the two drawers 50, i.e., the food stored in thesecond region 52 is visible when the drawers 50 are withdrawn.

In addition, the range of the horizontal angle according to the camera70 may include at least a portion of both ends of the shelf 40. Toacquire information about the food placed at the upper side of the shelf40, the camera 70 preferably captures a photo of a view including bothends of the shelf 40.

When the angle of view of the camera 70 changes, the range of theinterior of the storage compartment seen in the photo also changes.

FIG. 6 is a transverse cross-sectional view of FIG. 5, and FIG. 7 showsan image captured by the camera with the constituents positioned asshown in FIG. 6.

Hereinafter, a description will be given with reference to FIGS. 6 and7.

The inner case 12 is provided with an opening 14 to allow the user toaccess the storage compartment 22. The user may place food in thestorage compartment 22 or retrieve food from the storage compartment 22through the opening 14. The door 20 may open and close the opening 14.

The camera 70 may be installed outside of the drawer 50 to capturephotos while the drawer 50 is moving or stopping. Thereby, the camera 70may capture photos of the drawer 50 in the opened position, closedposition and moving state. That is, as the camera 70 continuouslycaptures photos, the photos captured by the camera 70 may present viewsof the drawer 50 at different positions according to the times at whichthe photos are captured.

In addition, the camera 70 may provide a screen including the lateststatus of storage and retrieval of items in the storage compartment 22.

Preferably, the camera 70 is installed within a range I between theopening 14 and one end of the shelf 40. In the case that a plurality ofshelves 40 are provided, the camera is preferably installed in a regionI (see FIG. 8) between one of the front edges of the shelves 40 locatedat the foremost position and the opening 14. That is, the camera ispreferably installed at the ceiling of the storage compartmentcorresponding to the region. Since the camera 70 needs to capture aphoto of the first region and second region 52 or a photo of the firstregion and third region 62, it cannot move farther toward the rear wallof the inner case 12 than the front edge of the shelf 40.

FIG. 6 shows a vertical angle of view of the camera 70. Preferably, oneend of the vertical angle of view is disposed such that even the frontend of the drawer 50 may be photographed when the drawer 50 is open.Additionally, the other end (rear edge) of the vertical angle of view ispreferably disposed such that even the rear edge of the uppermost shelfcan be photographed.

Since the second region 52 formed by the drawer 50 is also includedwithin the range of the vertical angle of view of the camera 70, thecamera 70 may allow the user to acquire information about the secondregion 52.

As shown in FIG. 7, the camera 70 may capture a photo including thefirst region 42 and the second region 52. At this time, the camera 70captures a photo of the two drawers 50 which are laterally disposed.Additionally, in the case that a plurality of shelves is provided, thefirst region 42 and the second region 52 which are divided by theshelves may be photographed together.

The camera 70 may be installed in the storage compartment 22. Thereby,it may capture a photo including information about the interior of thestorage compartment 22. Since the camera 70 is installed in the storagecompartment 22, it does not move. That is, the camera 70 is fixedlyinstalled in the storage compartment of the body which is generallyfixed at all times. Accordingly, the camera 70 is in a stationary statewhenever it captures a photo.

A photo captured by the camera 70 may include the first region 42 andsecond region 52 or the first region 42 and third region 62.

In the case that the camera 70 captures a photo including the firstregion 42 and the second region 52, the user may acquire informationabout the food stored at the upper side of the shelf 40 and the foodstored in the drawer 50 through the photo.

On the other hand, in case that the camera 70 captures a photo includingthe first region 42 and the third region 62, the user may acquireinformation about the food stored at the upper side of the shelf 40 andthe food stored in the barrier 60 through the photo.

The camera 70 may capture a photo of the interior of one storagecompartment 22 and transmit the captured photo, while the controller 100may acquire information about the plural storage spaces.

The camera 70 is disposed at the upper side of the first region 42,second region 52 and third region 62 and thus captures a photo whilelooking down.

FIG. 8 is a view illustrating a selection of a position of the camera.Hereinafter, a description will be given with reference to FIG. 8.

As shown in FIG. 8, it is possible to mount the camera at positions L1,L2, L3 and L4.

Position L1, which is outside the storage compartment 22, is an externalspace portioned by the outer case 10. In the case that the camera isinstalled at position L1, the view in front of the camera may beobstructed by the user when the user uses the refrigerator by accessingthe storage compartment 22 with the door opened.

In the case that the interior of the storage compartment is photographedby the camera installed outside the refrigerator as disclosed in theaforementioned Japanese patent document, the user may severely interferewith the view of the camera. Particularly, the drawer may be screenedfrom the camera, and thus movement of the drawer cannot be traced.Thereby, it may not be possible to recognize the image of the interiorof the drawer and movement of the drawer together, which is intended byone implementation. Additionally, position L1 may mean that the camerais positioned at the door. In this case, the camera moves duringphotographing since the door is a movable constituent. Accordingly, itmay be very difficult to obtain a stable and clear photo.

Position L2, which is in the storage compartment 22, represents theupper wall of the storage compartment 22, i.e., the ceiling. The cameraof this example is installed at position L2. With the camera arranged atthis position, the front view of the camera is not obstructed by theuser. In addition, wires for supply of power to the camera may beconveniently disposed since the upper wall of the storage compartment 22corresponds to the inner case 12. In addition, since the inner case 12is a constituent generally fixed at all times, it may be stably mountedto a constituent to which the camera is fixed.

Position L3, which is in the storage compartment 22, represents asidewall of the storage compartment 22. In this case, the camera isinclinedly installed at one side of the storage compartment 22 tocapture a photo of the opposing side of the storage compartment 22. Thecamera should be installed at a sidewall of the inner case 12 if it isto be disposed in the middle of the entire height of the storagecompartment 22 rather than being disposed at the uppermost end of thestorage compartment 22.

In the case that the camera is installed at one sidewall, the cameraneeds to be is installed such that the lens thereof faces the opposingsidewall. Accordingly, an image showing an asymmetric view of theinterior of the storage compartment 22 may be acquired due to differencebetween distances to the left side and the right side of the interior.

Position L4, which is in the storage compartment 22, represents one endof the shelf 40. To dispose the camera based on the height of thestorage compartment 22, the camera needs to be supported by a specificstructure. In this case, the camera may be fixed to one end of the shelf40.

In this case, the distance between the drawer 50 and the camera isshorter than the distance between the drawer 50 and position L2.Accordingly, a camera having a greater angle of view than when it isdisposed at position L2 may need to be used. In addition, an obtainedphoto may be severely distorted. Further, various problems causing userinconvenience such as installation of a wire connected to the camera onthe shelf 40 may be produced. In addition, the shelf 40 is a constituentthat is generally movable. Accordingly, when a food item is placed onthe shelf, the camera itself may be shaken.

The camera employed in one implementation may be a VGA camera having anangle of view of 120 degrees and providing performance of 20 fps.

FIG. 9 is a cross-sectional view showing main parts of the refrigerator.Hereinafter, a description will be given with reference to FIG. 9.

A basket 21 capable of storing food may be arranged at the inner side ofthe door 20.

The basket 21 may be installed on a surface of the door 20 facing thestorage compartment 22.

In this case, the basket 21 may accidentally appear in a photo capturedby the camera 70. Accordingly, in this example, the camera 70 maycapture a photo when it is sensed through the door sensor 120 that thedoor 20 has rotated an angle greater than a certain angle θ. To allowselection of a photo captured at a time point next to a time point atwhich sensing is implemented through the door sensor 120, informationabout time points may be transferred from the door sensor 120 to thecontroller 100.

For example, the certain angle θ may be between 60 degrees and 80degrees. The angle may change in a wide range. This angle may changeaccording to the capacity of the refrigerator, the size of the door, ora back-and-forth length of the basket 21.

The certain angle θ may be an angle at which the basket 21 does notappear in the photo captured by the camera 70. The certain angle θ mayvary depending upon the lateral width and back-and-forth length of thebasket 21.

The door switch 110 to sense whether or not the door 20 is opened may beinstalled at the upper side of the refrigerator. In this case, when thedoor switch 110 is pressed by the door 20, it may sense that the door 20closes the storage compartment. When the door switch 110 is not pressedby the door 20, it may sense that the storage compartment is opened.

A left hinge 300 to pivotably install the left door 20 on therefrigerator and a right hinge 320 to pivotably install the right door20 on the refrigerator may be provided. Each of the left hinge 300 andthe right hinge 320 may be divided into a portion provided with arotation axle coupled to the door 20 such that the door 20 rotates aboutthe axle, a portion installed at the outer case 10, and a connectingportion to connect the aforementioned two portions. That is, each of theleft hinge 300 and the right hinge 320 may be divided into a doorcoupling portion 300 c, 302 c, an outer case coupling portion 300 a, 320a, and a connecting portion 300 b, 320 b connecting the door couplingportion to the outer case coupling portion.

A left door sensor 120L to sense rotation of the left door 20 may beinstalled at the left hinge 300.

A right door sensor 120R to sense rotation of the right door 20 may beinstalled at the right hinge 320.

Each of the left door sensor 120L and right door sensor 120R mayindependently sense a rotational angle of a corresponding door.

FIG. 10 is a view illustrating operation of the door sensors.Hereinafter, a description will be given with reference to FIG. 10.

The door sensor 120 may include a light emitting unit 122 to radiatelight and a light receiving unit 124 to receive the light radiated fromthe light emitting unit 122,

The light radiated from the light emitting unit 122 may be reflected onthe upper surface of the door 20 and then transferred to the lightreceiving unit 124. In this case, to ensure stable reflection of light,the upper surface may be formed of a material having a highreflectivity.

In the case that the light radiated from the light emitting unit 122 istransferred to the light receiving unit 124, the door 20 may be sensedas having rotated by an angle less than the angle at which the doorsensor 120 is installed. That is, this may mean that the door 20 hasrotated by an angle less than the certain angle θ or that the door 20has not rotated from the position at which the door 20 closed thestorage compartment 22.

On the other hand, in the case that the light radiated from the lightemitting unit 122 is not transferred to the light receiving unit 124,the door 20 may be sensed as having rotated by an angle greater than theangle at which the door sensor 120 is installed. That is, this may meanthat the door 20 has rotated by an angle greater than the certain angleθ and thus the user is allowed to access the storage compartment 22.

To this end, as shown in FIG. 9, the left hinge 300 and the right hinge320 may be installed from the uppermost surface of the outer case 10 ofthe body to the lowermost surface of the door 20. Accordingly, when thedoor 20 rotates about the rotation center 302, 322, a portion (e.g., thestepped portion) of the uppermost door or the top surface of the doormay function as a light receiving part. The left door sensor 120L andthe right door sensor 120R may be installed at the connecting portions300 b and 320 b to function as a light emitting part. Due to thepositional relationship between the sensors 120L and 120R and the door,the sensors 120L and 120R may recognize the time at which the door isopened by an angle greater than or equal to the certain angle θ and thetime at which the door is closed to reach the certain angle θ.

FIG. 11 is a view specifically illustrating the left hinge unit andright hinge unit. For simplicity of illustration, the other constituentsof the refrigerator have not been shown.

Hereinafter, a description will be given with reference to FIG. 11.

The door sensor may be disposed at a position (where the user stands) infront of the front surface of the inner case, i.e., the opening.

A left door sensor 120L to sense rotation of the left door 20 may beinstalled at the left hinge 300.

A right door sensor 120R to sense rotation of the right door 20 may beinstalled at the right hinge 320.

The left door sensor or right door sensor may be disposed in an areabetween the opening and the rotation center of each of the hinge units.That is, the left door sensor or right door sensor may be disposed atthe connecting portion 300 b, 320 b of each hinge

The door sensor 120 may be connected to other constituents by aconnector c and a wire to receive electricity or to externally transfera signal. At this time, the door sensor 120 may be connected to thecontroller 100 to provide acquired signal information.

The left hinge 300 may be provided with a rotation center 302 aboutwhich the left door 20 rotates. In this case, the left door sensor 120Lmay be disposed on the right side of the rotation center 302. The spaceon the right side of the rotation center 302 is a space where the leftdoor 20, specifically, the top surface of the left door 20 may or maynot be positioned under the left door sensor 120L according to therotational angle of the left door 20. Accordingly, the left door sensor120L is preferably positioned at the connecting portion 300 b of thehinge 300. Thereby, when the size of the door or the size of the basketchanges, the certain angle θ may be flexibly varied. That is, the spaceon the right side of the connecting portion 300 b may be easily varied,and accordingly the position of the left door sensor 120L may be easilychanged at the connecting portion 300 b.

The wire w may be connected to the left door sensor 120L through therotation center 302. In addition, the wire w may also be connected tothe body of the refrigerator through the outer case 10.

The right hinge 320 may be provided with a rotation center 322 aboutwhich the right door 20 rotates. In this case, the right door sensor120R may be disposed on the left side of the rotation center 322. Thespace on the left side of the rotation center 322 is a space where theright door 20 may or may not be positioned under the right door sensor120R according to the rotational angle of the right door 20. For thesimilar reason, the right door sensor 120R is preferably installed atthe connecting portion 320 b of the hinge 320.

The time sensing is conducted by the door sensor 120 and may be the timeat which a photo of each of the regions in the storage compartment 22,specifically, the first region 42 or the third region 62 is captured.That is, the time sensing is conducted by the door sensor 120 and may bethe time at which a photo of a storage space outside the drawer 50 iscaptured.

Meanwhile, since the door sensor 120 radiates and receives light, itemits light downward. The light radiated downward is reflected upward bythe doors 20.

The door sensor 120 and the door 20 may be maintained at a shortdistance from each other to allow the door sensor 120 to receive asufficient amount of light. At this time, the maximum vertical distancebetween the door sensor 120 and the door 20 may be 20 mm.

FIGS. 12A to 12C show exemplary screen images provided to the user.Hereinafter, a description will be given with reference to FIGS. 12A to12C.

In the example of FIG. 12A, the first region 42, second region 52 andthird region 62 are all provided to the user. In the example of FIG.12B, the second region 52 and third region 62 are provided to the user.In the example of FIG. 12C, the first region 42 and second region 52 areprovided to the user.

A description will now be given with reference to 12A.

One photo captured by the camera 70 may be divided into a first imageshowing the first region 42, a second image showing the second region52, and a third image showing the third region 62. That is, one photo ora plurality of photos may be divided based on the respective regions.Herein, the first image, second image and third image may be images cutfrom the captured photo according to the range of the respective regionsshown in the images.

The divided photos may be independently provided to the user, as shownin FIG. 12A. That is, a photo of each region may be individuallyprovided to the user such that the user easily acquires informationabout the food stored in each region.

The display 14 may be disposed such that a plurality of storage regionsis shown in a frame.

That is, the second image showing the second region 52 is disposed onthe upper left side and upper right side in one rectangular frame, andthe third image showing the third region 62 is disposed on the lowerleft side and lower right side in the rectangular frame.

The images of the regions may be disposed in one frame on the display 14such that the images have the same width. To this end, the controllermay cut the captured photo and correct the size of the photo to providethe photo to the display 14. At this time, the corrected image is shownat the position corresponding to the frame.

At least one portion of the images provided to the display may becorrected to have the same width to produce a two dimensionalarrangement making the user feel like they are viewing the configurationof the storage compartment seen by the user when the user actually opensthe doors of the refrigerator.

The display 14 may provide two overlapping regions through one screen,thereby providing the user with information about food items which arepositioned not to be seen at a glance.

Specifically, the second image and the third image show the spacesdisposed in an overlapping manner with the drawer withdrawn.Accordingly, the user may not acquire information about the two storagespaces simultaneously with the drawer withdrawn. However, the display 14may provide the information about the two storage spaces simultaneously.

The controller 100 may update (replace) the respective imagesindividually to provide the user with the information about the kinds offood accommodated in the respective regions.

For example, in the case that a portion corresponding to the secondimage needs to be upgraded through the photo captured by the camera anda portion corresponding to the third image does not need to be upgraded,the second image alone may be replaced with a new one.

On the other hand, in the case that the portion corresponding to thethird image needs to be upgraded through the photo captured by thecamera and the portion corresponding to the second image does not needto be upgraded, the third image alone may be replaced with a new one.

The second image and the third image in one frame may be updatedindependently of the other images.

Of the two second images disposed on the left and right sides, thesecond image disposed on the left side alone may be updated or thesecond image disposed on the right side alone may be updated. In thiscase, each of the third images may be individually updated.

That is, the display 14 may provide a frame allowing a plurality ofimages to be provided therein to provide information about items storedin the storage compartment, and multiple images may be individuallyupdated independently of the other images. Particularly, since thesecond image and the third image are from different photos, thecontroller may determine the regions represented by the photos andupdate the images.

Herein, the width of the second image may be equal to or substantiallysimilar to the width of the third image. That is, the portionscorresponding to the second image and the third image may be correctedby the controller to have the same width and then provided to thedisplay.

In the case that the size of the second image is different from that ofthe third image, the second image and the third image may be selectedfrom a photo such that the second image and the third image havedifferent numbers of pixels, i.e. different sizes with respect topixels. Alternatively, the second image and the third image may beselected to have the same number of pixels but may be corrected by thecontroller to have different sizes when they are presented on thedisplay 14.

The second image and the third image may be vertically disposed toprovide a view of a vertical arrangement in one plane. The region shownin the second image is actually disposed on the region shown in thethird image. Accordingly, the user may intuitively recognize the imagesprovided through the display and thus easily understand the informationprovided through the images.

The display 14 may provide the user with a screen with both the firstimage and third image replaced. Alternatively, it may provide the userwith a screen showing the existing first and third images and a replacedsecond image.

In FIG. 12A, the food storage compartment shelf may represent the firstregion 42, the vegetable compartment may represent the second region 52,and the multiaccommodation corner may represent the third region 62.Depending upon the user preference and intention of the image provider,the configuration may take various forms.

The screen shown in FIG. 12A may be provided not only to the display 14installed in the refrigerator, but also to a separate external terminaldevice other than the refrigerator such as, for example, a smartphone.Accordingly, when the user is away from home, the user may obtaininformation about the food stored in the refrigerator and use theinformation in shopping. Accordingly, the user may understand the kindsof food items stored in the refrigerator without opening the door. Thescreen provided to the user may display one storage compartment which isdivided into a plurality of storage regions. Accordingly, the user mayeasily acquire information about the food items stored in one storagecompartment classified based on the positions of the food items in thestorage compartment.

The image arranged in the position of the vegetable compartment shows aregion vertically extending from the front edge of the drawer 50 to thefront edge of the shelf 40 with the drawer 50 withdrawn. The imagearranged in the position of the multiaccommodation corner shows a regionoverlapping the drawer 50 arranged in the introduced position with thedoor 20 positioned not to obstruct the view. The region at the upperside of the shelf 40 is shown in the area of the aforementioned shelf.

According to one implementation, the screen provided to the user doesnot show one storage compartment as it is, but shows separated imagesdisposed on divided areas. Therefore, the user may easily acquireinformation. In other words, rather than using the picture shown in FIG.7, the respective regions may be separated from one picture andindividually and independently used.

A screen having a different configuration than that of the FIG. 12A maybe provided as shown in FIG. 12B. In this case, the camera 70 maycapture a photo containing the second region 52 and the third region 62.Alternatively, the camera 70 may capture a photo containing the firstregion 42, second region 52 and third region 62, but may also providethe user with images corresponding to only two of the regions.

Another screen having a different configuration as shown in FIG. 12C maybe provided. In this case, the camera 70 may capture a photo containingthe first region 42 and the second region 52. Alternatively, the camera70 may capture a photo containing the first region 42, second region 52and third region 62, but may provide the user with images correspondingto only two of the regions.

That is, according to one implementation, a photo containing the threeregions may be captured and then only relevant images may be providedfor the user. However, it may be possible to provide the user with animage corresponding to at least the internal storage region of thedrawer, i.e., the second region and an image corresponding to anotherregion together.

FIG. 13 is a view illustrating adjustment of a photo captured by thecamera.

Hereinafter, a description will be given with reference to FIG. 13.

The screen image shown in FIG. 13 may be an example of screens displayedon the display 14. In the case that the display is not provided to therefrigerator, the screen of FIG. 13 may be a screen displayed on aseparate display device of an operator when the refrigerator isfabricated. In addition, the screen of FIG. 13 may represent a screendisplayed on the user's cell phone.

In the case that the camera 70 is provided, the camera 70 may not beaccurately installed at a desired position due to assembly tolerance,

Accordingly, in one implementation, it is possible for the operator oruser to change an adjustment line 15 dividing the regions through aphoto or image shown by the camera 70.

The adjustment line 15 may include a vertical adjustment line and ahorizontal adjustment line. The vertical adjustment line may be disposedat the center and opposite ends of an image. The user may use buttonsshown on the screen to vertically move the vertical adjustment line andhorizontally move the horizontal adjustment line. Thereby, the user mayselect a portion of the image cut and provided to the display 14.

Meanwhile, limit lines 16 which limit the range of movement of theadjustment line 15 may also be shown. In this case, the limit lines 16may be disposed horizontally and spaced a predetermined distance fromeach other with the vertical adjustment line placed therebetween. Inaddition, the limit lines 16 may be disposed vertically and spaced apredetermined distance from each other with the horizontal adjustmentline placed therebetween

Given the limit lines 16, installation of the camera 70 may bedetermined to be poor and the position where the camera 70 is installedmay be adjusted, in the case that a desired image is not acquired evenwhen the adjustment line 15 is moved.

In the case that two drawers are provided in the storage compartment asshown in FIG. 13, the vertical adjustment line may be disposed betweenthe drawers.

In addition, the horizontal adjustment line may be disposed at the frontedge of the shelf.

With movement of the horizontal adjustment line and the verticaladjustment line completed, an image showing the left portion of thesecond region or third region may be configured on the lower left sideof the point of intersection between the horizontal adjustment line andthe vertical adjustment line. In this case, the image may horizontallycover a particular number of pixels from the horizontal adjustment lineand vertically cover a particular number of pixels from the verticaladjustment line.

An image showing the right portion of the second region or third regionmay be configured on the lower right side of the point of intersectionbetween the horizontal adjustment line and the vertical adjustment line.In this case, the image may horizontally cover a particular number ofpixels from the horizontal adjustment line and vertically cover aparticular number of pixels from the vertical adjustment line.

The first region may be configured at the upper side of the horizontaladjustment line. In this case, the image may vertically cover aparticular number of pixels from the horizontal adjustment line.

By separately displaying the images of the left and right portions ofthe third region corresponding to the vegetable compartment as shown inFIG. 13, whether a photo corresponding to the drawer has been acquiredmay be checked.

Once adjustment by the user or operator is completed, the InputCompleted button may be clicked to input an indication that the movementof the adjustment line 15 has been completed. Therefore, when theinstallation error of the camera is small, the positions of the verticalline and/or horizontal line dividing the regions may be adjusted in thetaken picture. Thereby, portions representing a specific region may beclearly displayed. When there are plural regions divided vertically orhorizontally, a plurality of horizontal adjustment lines of verticaladjustment lines may be provided.

FIG. 14 is a perspective view showing the camera, FIG. 15 is a viewillustrating main parts of the camera, and FIG. 16 is a cross-sectionalview of FIG. 14. Hereinafter, a description will be given with referenceto FIGS. 14 to 16.

The camera 70 may include a camera module 71 including a camera lens 71a and a body 71 b and cases 73 and 74 where the camera module 71 isaccommodated.

The camera module 71 may include an image sensor to convert imageinformation transmitted through the camera lens 71 a into a digitalsignal.

The cases may include a first case 73 and a second case 74 coupled tothe first case 73 to internally form a predetermined space. The body 71b and the camera lens 71 a may be accommodated in between the first case73 and the second case 74. Coupling between the first case 73 and thesecond case 74 may be maintained with a separate sealing member placedbetween the first case 73 and the second case 74 to prevent introductionof moisture or dust into the first case 73 and the second case 74.

Since the body 71 b and the camera lens 71 a are inserted into the spaceformed by the first case 73 and the second case 74, they are preventedfrom contacting the foaming agent filling the space between the innercase 12 and the outer case 10. In addition, the camera module 71 may beprotected by being prevented from directly contacting the cooled airaccommodated in the storage compartment 22.

The first case 73 may include a transparent window 80 installed at thefront edge of the camera lens 71 a and a heater 84 providing heat to thetransparent window 80.

Preferably, the transparent window 80 may be formed of a transparentmaterial.

The heat provided by the heater 84 may prevent dew condensation on thetransparent window 80.

In the case that the heater 84 is installed at the camera lens 71 a toprevent dew condensation, the camera lens 71 a may be deformed. In thiscase, possibility of occurrence of aberration and permanent damageresulting from the deformation may increase. Accordingly, in thisexample, the transparent window 80 is arranged not to directly contactthe camera lens 71 a.

The transparent window 80 may be formed of any of plastics and glass toacquire images from the camera. However, the thermal conductivity ofplastics is low. Accordingly, in the case that plastics are used to formthe transparent window 80, power consumption may increase relative tothe case in which glass is used to form the camera 70. Therefore, thetransparent window 80 is preferably formed of glass to reduce powerconsumption.

If the thermal conductivity increases, heat supplied from the heater 84may be easily conducted to portions of the transparent window 80 whichdo not directly contact the heater 84. For example, if the thermalconductivity decreases. The temperature may be high at a portion of thetransparent window 80 adjacent to the heater 84 and be low at a portionfar apart from the heater 84. In this case, it may be difficult toprevent dew condensation. Moreover, more thermal energy needs to besupplied by the heater 84, and accordingly energy efficiency may belowered.

Meanwhile, since the heater 84 makes direct surface-contact with thetransparent window 80, heat transfer efficiency may be improved.

The heater 84 may be disposed at the opposite side of the surface of thetransparent window 80 exposed to the storage compartment 22.Accordingly, the heat generated by the heater 84 may be prevented frombeing supplied to the storage compartment 22 without being transmittedthrough the transparent window 80.

Particularly, the heater 84 may include a hotwire coil to generate heat.The hotwire coil may make surface contact with the transparent window80.

In addition, the transparent window 80 and the camera lens 71 a arepreferably spaced a predetermine gap g from each other, in considerationof deformation of the transparent window 80 by heat supplied thereto andassembly tolerance.

In addition, the camera 70 may include a cable 79 to receive electricitywhich is externally supplied and transmit a signal related to anacquired photo. The cable 79 may electrically connect the camera module71 to an external constituent.

Meanwhile, the case is provided with a first seating segment 75 capableof fixing the camera 70 to another constituent. The first seatingsegment 75 may be provided to any one of the first case 73 and thesecond case 74.

The first seating segment 75 may include a surface having apredetermined area and may be disposed at a desired position andinclined at a desired angle.

The first seating segment 75 may be provided to both sides of the camera70.

Thereby, both sides of the camera 70 may be fixed by an externalconstituent.

The second case 74 may be provided with a second seating segment 76 orsecond seating surface 76 having a predetermined area and formed in astepped manner. The second seating segment 76 is provided at a positiondifferent from the position of the first seating segment 75 to allow thecamera 70 to be stably coupled to another constituent at a specificposition at a predetermined angle. Therefore, the camera 70 may be fixedat the predetermined angle at at least three support points by the firstseating segments 75 and the second seating segment 76 with respect toother constituents. The supporting structure of the camera 70 will bedescribed in detail later.

A VGA resolution may be applied to the camera 70. The camera 70 mayadopt the VGA resolution (640 by 480) of 0.3 mega pixels.

The interface of the camera 70 may be of a USB type and driven using theUSB power of 5V and 500 mA.

The value of current supplied to the camera 70 may be 87 mA in thestandby mode and 187 mA (for the heater, 67 mA) in the active mode.Power consumption may be in proportion to the supplied current.Operation of the camera 70 may be divided into the standby mode and theactive mode, and power may always be supplied to the heater 84 toprevent dew condensation on the surface of the transparent window 80.

When a rated voltage is applied to the camera 70, the camera 70 operatesin the standby mode. When the door switch 110 senses that the door 20 isopen, the camera 70 is switched from the standby mode to the active modeto take a picture. That is, when power is applied to the refrigerator,the camera may operate in the standby mode. When the door switch 110senses that the door 20 is open, the camera 70 may be switched to theactive mode to take a picture.

FIG. 17 is a view illustrating disposition of the heater. Hereinafter, adescription will be given with reference to FIG. 17.

The camera 70 may have a horizontal angle of view of 115° a verticalangle of view of 95°, and a diagonal angle of view of 145°. That is, thecamera 70 may be disposed such that the horizontal angle of view may begreater than the vertical angle of view. Through this disposition, aphoto providing food information about the storage compartment 22 may beacquired in the case that the length of the storage compartment 22 inthe width direction of the storage compartment 22 (the horizontaldirection in FIG. 17) is greater than the length of the storagecompartment 22 in the front-to-back direction (the vertical direction inFIG. 17).

Meanwhile, the heater 84 may be formed along the outline of thetransparent window 80. In this case, the heater 84 may be disposed in aquadrangular shape so as not to be placed within the angles of view ofthe camera 70.

In the case that the heater 84 has a quadrangular shape, the heat may besupplied to a relatively wide area of the transparent window 80.Accordingly, energy efficiency of the heater 84 may be improved.

Particularly, in the case that a gap g is formed between the camera lens71 a and the transparent window 80 as shown in FIG. 16, the angles ofview of the transparent window 80 form a rectangular shape as shown inFIG. 17. In the case that the transparent window 80 and the camera lens71 a contacts without the gap g present therebetween, the heater 84 maybe formed in the shape of a square with sides each of which has a lengthgreater than or equal to the diameter of the camera lens 71 a since theheater 84 contacts one surface of the transparent window 80.

However, in this example, the gap g is present and the vertical angle ofview is different from the horizontal angle of view as described above.Accordingly, the heater 84 is preferably formed in the shape of arectangle having one side longer than another side.

FIGS. 18A to 18C are views showing photos captured with the cameramounted to the refrigerator such that the camera is vertically inclined,and FIGS. 19A to 19C are views showing photos captured with the cameramounted to the refrigerator such that the camera is horizontallyinclined. Hereinafter, a description will be given with reference toFIGS. 18A through 19C.

FIG. 18A shows a photo captured with the camera turned upward by 5degrees, FIG. 18B show a photo captured with the camera installed at adesired position, and FIG. 18C shows a photo captured with the cameraturned downward by 5 degrees. The photos shown in FIGS. 18A to 18C mayprovide the user with information about the food stored in the storagecompartment. However, in the case that the camera is turned by an angleexceeding the angles shown in FIG. 8, desired information may not beprovided for the user.

FIG. 19A shows a photo captured with the camera turned leftward by 4degrees, FIG. 19B show a photo captured with the camera turned leftwardby 2 degrees, and FIG. 19C shows a photo captured with the camerainstalled at a desired position.

Compared to the photos of FIGS. 18A to 18C which are captured with thecamera installed to be turned in the vertical direction (back-and-forthdirection), the photos captured with the camera which is not installedat a desired horizontal (lateral) angle provided images which arerelatively significantly distorted images.

As can be seen from FIGS. 18A through 19C, even a delicate difference inthe angle of installation of the camera may result in significantdistortion of a captured photo. Therefore, the angle of installation ofthe camera needs to be controlled such that the produced assemblytolerance is as small as possible. A fine error may be adjusted throughthe adjustment line described in FIG. 13, and thus optimum pictures maybe provided.

FIG. 20 is a view illustrating a camera housing in an assembled state,FIG. 21 is a front view showing a first housing, and FIG. 22 is a frontview showing a second housing. FIG. 23 is a front view showing the firsthousing with the camera installed, FIG. 24 is a transversecross-sectional view of FIG. 23, and FIG. 25 is a cross-sectional viewillustrating the cameral housing installed at the inner case.Hereinafter, a description will be given with reference to FIGS. 20 to25.

A cameral housing may be provided to install the camera 70 in the innercase 12. That is, a camera housing to receive the camera shown in FIG.14 and to fix the camera to the ceiling of the storage compartment maybe provided.

The camera housing may include a first housing 400 coupled to the lowersurface of the ceiling of the inner case 12, a second housing 410coupled to the upper surface of the ceiling of the inner case 12, and athird housing 420 and fourth housing 430 coupled to the first housing400. Herein, the first housing 400 may be coupled to the top surface ofthe ceiling of the storage compartment

The third housing 420 may close the front of the camera 70 such that allthe parts of the camera 70 except the transparent window 80 are notdirectly exposed to the door 20. That is, the third housing 420 mayfunction as a cover that covers the front surface of the camera 70. Inthis case, the third housing 420 and the fourth housing 430 may beconstructed with one constituent. The fourth housing 430 may be a decohosing interposed between the third housing 420 and the first housing.

The camera 70 may be installed at the first housing 400 such that thedegree of lateral inclination (horizontal inclination) andback-and-forth inclination (vertically inclination) of the camera 70 aremanaged.

As shown in FIG. 21, the first housing 400 may be provided with aseating part 402 on which the first seating segment 75 of the camera 70is seated. Herein, two first seating parts 402 may be provided to allowthe left and right first seating segments 75 to be fastened to each ofthe first seating parts 402.

The first seating segment 75 may be provided to both left and rightsides of the camera 70 such that the first seating segment 75 isinclined at a predetermined angle. Thereby, the camera 70 may bedisposed to be inclined at a desired angle when it is installed at theinner case 12.

The first seating part 402 may be provided with a hole 404. Thereby, thefirst seating segment 75 and the first seating parts 402 may be fixed bya screw. The first seating parts 402 are arranged in horizontaldirection such that they are disposed at different heights. Thereby, thecamera 70 may be stably fixed.

The first housing 400 may be provided with a second seating part 408allowing the second seating segment 76 of the camera 70 to be seatedthereon. Herein, the second seating part 408 may be formed to have apredetermined area. Thereby, it may fix the degree of inclination of thecamera 70 while making surface-contact with the second seating segment76.

That is, the camera 70 may be coupled to the first housing 400 with aplurality of surfaces thereof contacting the first housing 400. Thereby,the camera 70 may function to fix the camera 70 at a position inclinedat a desired angle. In other words, it may be fixed to have at leastthree support points and supported at a predetermined angle with respectto the first housing 400. It may be fixed at a horizontal angle to beparallel with the first housing 400 and the vertical angle may be set tobe a predetermined angle.

The first housing 400 may be provided with a fastening hole 406 to becoupled to the second housing 410 by a screw.

As shown in FIG. 22, the second housing 410 is provided with a contactsurface 416 capable of making surface-contact with the upper surface ofthe ceiling of the inner case 12. Herein, the contact surface 416 isdistributed over a wide area, generally forming quadrangular shape.Thereby, it may allow the second housing 410 to be stably seated on theupper surface of the ceiling of the inner case 12. Therefore, thecontact surface 416 may have a surface parallel with the ceiling of theinner case 12. Thereby, the reference line or plane for the installationangle of the camera may be identical to the ceiling surface of the innercase 12.

The second housing 410 may be provided with an accommodation space 412to accommodate the cable 79 of the camera 70. A through hole 418 may beformed at one side of the accommodation space 412 to allow a wireconnected to another constituent of the refrigerator to be exposed tothe accommodation space 412. At this time, the wire connected to anotherconstituent of the refrigerator may be connected to the cable 79 in theaccommodation space 412. Accordingly, electricity may be supplied to thecamera 70 through the cable 79, and a signal for a photo captured by thecamera 70 may be transferred to the connected constituent.

The second housing 410 may be provided with a fastening part 414 coupledto the first housing 400. The fastening part 414 may be coupled to thefastening hole 406 by a screw. It is preferable that an assemblytolerance is not produced when the fastening part 414 is coupled to thefastening hole 406. Accordingly, the fastening part 414 may be formed toprotrude to a predetermined height to facilitate surface-contact withthe fastening hole 406.

As shown in FIGS. 23 and 24, when the camera 70 is installed at thefirst housing 400, the camera 70 may be disposed to be inclined at apredetermined angle. As shown in FIG. 24, one side surface 400 a of thefirst housing 400 may be formed in the shape of a flat surface tocontact the inner case 12. The camera 70 is inclined at a predeterminedangle with respect to one surface of the first housing 400. In addition,the one side surface 400 a is coupled to the lower surface of theceiling of the inner case to be parallel with the contact surface 416 ofthe second housing 410. Therefore, as the camera is coupled to the firsthousing 400 and the first housing 400 is coupled to the second housing420, the error of installation angle of the camera 70 with respect tothe ceiling surface of the storage compartment may be significantlyreduced.

Referring to FIG. 25, the second housing 410 is disposed to be exposedto the space over the inner case 12, i.e., the space between the innercase 12 and the outer case 10 which is filled with a foaming agent. Thesecond housing 410 is disposed outside the storage compartment 22, butis disposed in the space defined by the outer case 10. The position ofthe second housing 410 may be temporarily fixed by adhesively attachingthe second housing 410 to the inner case 12. Thereafter, it may befinally fixed by the foaming agent filling the space between the innercase 12 and the outer case 10.

The first housing 400 and the second housing 410 may be coupled to eachother with the inner case 12 placed therebetween. Since the firsthousing 400 is coupled to the second housing 410 with the second housing410 fixed at a particular position on the inner case 12, the position ofthe camera 70 may be fixed.

With the camera 70 inclined at a particular angle with respect to thefirst housing 400, the third housing 420 and the fourth housing 430 maybe coupled to the second housing 410. As described above, the thirdhousing 420 and the fourth housing 430 may be constructed with oneconstituent, and thus only a constituent functioning as the fourthhousing 430 may be coupled to the first housing 400.

The fourth housing 430 may allow only the transparent window 80 of thecamera 70 to be exposed to the storage compartment 22 and may not allowthe other part of the camera 70 to be exposed. Accordingly, moisture,dust and cooled air present in the storage compartment 22 may beprevented from affecting the camera 70.

The structure of the housing of the camera and the installationdirection and position of the housing on the inner case have a purposeof reducing dew condensation on the transparent window 80 in a technicalaspect. A relevant detailed description will be given later.

FIG. 26 shows a table comparing power consumption of a camera accordingto one example with a counterpart example, and FIGS. 27A and 27B areviews comparing the supplied standby current with the drive current.FIG. 27A shows currents supplied to a camera of a counterpart example.FIG. 27B shows currents supplied to a camera according to oneimplementation. Hereinafter, a description will be given with referenceto FIGS. 26 through 27B.

In the counterpart example, a standby current of 70 mA is supplied tothe camera 70.

At the time the door is opened, 50 mA is additionally supplied, and thusa drive current of 120 mA is supplied to the camera 70. In thecounterpart example, if the door is opened twenty five times a day, thecamera 70 consumes 253 Wh a month. If the door is opened fifty times aday, the camera 70 consumes 254 Wh a month.

On the other hand, according to one implementation, a standby current of20 mA is supplied to the camera 70. At the time the door is opened, 100mA is additionally supplied, and thus a drive current of 120 mA issupplied to the camera 70. If the door is opened twenty five times aday, the camera 70 consumes 74 Wh a month. If the door is opened fiftytimes a day, the camera 70 consumes 76 Wh a month.

According to the two experimental results, it can be seen that the powerconsumed by the camera 70 may be reduced by about 70% by reducing thestandby current. Accordingly, it has been found that the power consumedby the camera 70 may be significantly reduced by reducing the standbycurrent, on the condition that the same drive current is supplied. Whenthe refrigerator continuously operates around the clock, the time forwhich the door is opened is relatively short. Thus, if the value ofcurrent in the active mode is constant, power consumption may be reducedin reducing the current applied to the camera in the standby mode.

In the case that the standby current is reduced as shown in FIGS. 27Aand 27B, the time at which the drive current is supplied to the camera70 is delayed. This is because difference between the standby currentand the drive current increases delay time. In other words, when thestandby mode is switched to the active mode, photographing is notimmediately performed until a predetermined delay time elapses.

Accordingly, the time at which the camera 70 actually captures a photois delayed. If the time lag between when a command to capture a photo istransferred to the camera 70 and when the camera 70 captures a photo islong, the photo actually captured by the camera 70 may fail to providedesirable information about food to the user.

In other words, as the standby current decreases, the power consumed bythe camera 70 may be reduced, but a delay time taken to increase thecurrent to the drive current for driving of the camera 70 may becomelong. Accordingly, by calculating the usual time taken to capture aphoto at a desired time point, the value of the standby current ispreferably selected so as to shorten the delay time below theaforementioned time. For this reason, in this example, the current ispreferably increased step by step from the standby current value to thedrive current value. That is, sufficient delay time is provided when thecurrent increases the relatively low standby current value to the drivecurrent value. That is, rapid increase of current may be prevented.Thus, power consumption may be reduced and stability of the camera maybe secured.

Accordingly, in this example, the camera 70 is controlled to be drivenat the time the door 20 is opened. That is, when the door switchdetermines that the door is opened, the standby mode is switched to thedrive mode. However, a valid picture of the pictures taken by the cameraare captured after a predetermined time elapsed after the door isopened. This is because a few seconds may take for the user to updatethe food items. That is, even if delay time becomes long, a photo may besurely captured at a desired time point. Therefore, when the camera isswitched from the standby mode to the drive mode, photographing may beperformed by the result of sensing by the door switch. In addition, atthe reference time at which a valid picture is selected, photographingmay be performed by the result of sensing by the door sensor.

FIG. 28 is a view illustrating the start time of photographing by thecamera and continuous photographing by the camera. Hereinafter, adescription will be given with reference to FIG. 28.

In this example, the door switch may be provided as shown in FIG. 4.

When the door switch 110 senses that the storage compartment 22 isopened, the controller 100 may command the camera 70 to start to capturea photo. That is, when the storage compartment 22 is opened by movementor rotation of the door 20, the camera 70 may start to capture a photo.

On the other hand, when the door switch 110 senses that the storagecompartment 22 is closed, the controller 100 may command the camera 70to stop capturing a photo. That is, when the storage compartment 22 isclosed by movement or rotation of the door 20, the camera 70 mayterminate the photographing operation.

As shown in FIG. 28, when electricity is supplied from an external powersource to the refrigerator, a standby current is supplied to the camera70 (S1). At this time, the camera 70 may be driven in the standby mode.In the standby mode, the camera 70 does not capture a photo.

The door switch 110 may sense the opening of the door 20 (S10).

Upon determining that the door 20 opens the storage compartment 22, thecamera 70 may start to capture a photo (S14). Particularly, the camera70 may continuously capture photos at predetermined time intervals.

At this time, a current greater than the current supplied in the standbymode is supplied to the camera 70, and thus the camera 70 may beoperated in the drive mode. To supply a current greater than the standbycurrent, i.e. the drive current to drive the camera 70, a predeterminedtime, i.e. a delay time may be inevitably produced. That is, to increasethe magnitude of the current supplied to the camera 70, a certain amountof time is actually needed. When a predetermined time, delay timeelapses after the current for implementation of the drive mode issupplied to the camera 70, the camera 70 may capture a photo with thedrive current supplied thereto.

Since the camera 70 is switched to the drive mode for photographing fromthe time the door 20 is opened, there is little risk of capturing aphoto earlier and failing to capture a photo at a particular time. Thisis because photographing time is delayed due to the delay time producedwhen the standby current is supplied before the desired photographingtime and the value of current is increased a the desired time.

In addition, the number of photos captured per second may be reduced. Ifthe camera 70 captures thirty photos per second, it is rarely possiblethat a change occurs in the storage compartment 22 for 1/30 second,i.e., a time interval at which a photo is captured. The interior of thestorage compartment 22 may be changed by an action performed the user,but it is rarely possible that an average person terminates an actionfor the storage compartment 22 in 1/30 second. That is, a time intervalat which a photo is captured by the camera 70 may be relativelyextended. Therefore, a camera 70 of a relatively low price may be used,and accordingly, production cost of the refrigerator may be reduced.

The photos captured by the camera 70 may be stored in the storage unit18 (S16). In addition, in the case that the number of photos capturedper second by the camera 70 is reduced, the number of photos stored inthe storage unit 18 is also reduced, and thus the storage unit 18 may bemanaged more efficiently.

Meanwhile, the door sensor 120 may sense whether the door 20 has rotateda particular angle (S18).

In the case that the door sensor 120 does not sense rotation of the door20, photos stored in the storage unit 18 before a predetermined time maybe deleted (S90). That is, photos which are not processed to be providedfor the user may be deleted to prevent the amount of stored objects fromincreasing.

On the other hand, in the case that the door sensor 120 determines thatthe door 20 has rotated the particular angle, it selects a photocaptured at a corresponding time (S92). At this time, the selectedpicture, which is a valid picture, may be processed by the controller100 and provided for the user. For example, the photo processed by thecontroller 100 may be transferred to the display 14.

A certain amount of time is needed for the user to rotate the door 20 toaccess the storage compartment 22. The delay time elapses for thecertain amount of time, and thus a photo may be captured at the time thedoor 20 has rotated the particular angle.

Accordingly, the controller 100 may select a photo at a desired time.

In the case that the camera 70 is switched, at the time point at whichthe door sensor 120 senses the rotation, from the standby mode to theactive mode to capture a photo, the camera 70 may undesirably capture aphoto of the storage compartment only when a delay time elapses afterthe time point.

Since the camera 70 begins to capture a photo at the time the door 20 isopened, a standby current of a relatively low value may be supplied tothe camera 70. In the case that a standby current of a relatively lowvalue is supplied, the delay time taken to supply the drive current mayincrease, but the camera 70 starts photographing earlier in a controlflow. Accordingly, failing to capture a photo at a desired time may beprevented.

FIG. 29 is a view illustrating the start time of photographing by thecamera and continuous photographing by the camera. Hereinafter, adescription will be given with reference to FIG. 29.

In FIG. 29, a time to select a photo does not depend on rotation of thedoor 20, unlike the example illustrated in FIG. 28. Rather, the time toselect a photo is determined according to the position of the drawer 50.The other details are the same as those of the example of FIG. 28, andthus they will not be discussed again for simplicity of illustration.However, manipulation of the drawer 50 may presume opening of the door20. The reference time at which a specific picture is selected may bedetermined by the door sensor. The reference time at which a specificpicture of the specific region in the specific picture may be determinedaccording to the determination by the door sensor and the determinationby the drawer sensing unit 130. Therefore, in any cases, to determinethe reference time for the valid picture, determination by the doorsensor is needed.

The user may withdraw the drawer 50 to store food in the drawer 50 or toretrieve stored food from the drawer 50. At this time, the user maywithdraw the drawer 50 to a sufficient distance to use the drawer 50.

Upon completing use of the drawer 50, the user may push back the drawer50. The time at which this event occurs may represent the particularstate disclosed in S19.

The camera 70 starts to capture a photo from the time the door switch110 senses opening of the door 20. Accordingly, when the drawer 50 isput in the particular state, a photo captured at the corresponding timemay be selected and provided for the user (S92).

When the door 20 is opened, the camera 70 captures a photo, and movementof the drawer 50 may be sensed from the time the photo is captured,i.e., the time when the door 20 is opened.

FIG. 30 is a view illustrating a drawer sensor according to one example,and FIGS. 31A and 31B are views illustrating a method for the drawersensor of FIG. 30 to sense movement of the drawer. Hereinafter, adescription will be given with reference to FIGS. 30 to 31B.

Referring to FIG. 30, it may be possible to configure the drawer sensingunit 130 as a separate constituent rather than providing the drawersensing unit 130 to the controller 100.

The drawer sensing unit 130 may be provided with one light emitting partand two light receiving parts. That is, by analyzing the time when thelight radiated from the light emitting part is incident on the two lightreceiving parts, the direction of movement of the drawer 50 may besensed.

A plurality of slits 134 may be continuously disposed on the drawer suchthat the light radiated from the light emitting part of the drawersensing unit 130 and incident on the slits 134 are reflected. Thereby,the light may be sequentially incident on the two light receiving parts.

In FIGS. 31A and 31B the x-axis represents time, and the y-axisrepresents signals generated at the time when the light is received bythe two light receiving parts Photo TR1 and Photo TR2. That is, in FIG.31A, Photo TR1 receives light earlier than the Photo TR2.

In FIG. 31B, Photo TR2 receives light earlier than Photo TR1.

The view of FIG. 31B may represent a moving state of the drawer 50 inwhich the user withdraws the drawer 50 to use the drawer 50, and theview of FIG. 31B may represent a state of the drawer 50 in which theuser introduces the drawer 50 into an original position after finishingusing the drawer 50.

Accordingly, when the drawer sensing unit 130 senses signals as in FIG.31A, the controller 100 may determine that it is time to acquire a photocaptured by the camera 70 and select the photo as a photo of theinterior of the drawer 50, i.e., the third region at the correspondingtime. That is, according to the result of sensing or determination bythe drawer sensing unit 130, the time at which a picture correspondingto the third region may be acquired may be determined. The portioncorresponding to the third region and the other portions in the acquiredpicture may be independently and individually selected, separated andupdated.

FIG. 32 is a view showing markers indicated on the drawer. Hereinafter,a description will be given with reference to FIG. 32.

Unlike FIGS. 30 to 31B, movement of the drawer 50 may be sensed using amarker, which is a specific indication, for the drawer 50, withoutemploying a separate machine or electronic device.

The drawer 50 may be provided with a marker having various patterns(e.g., a pattern of white and black colors). The camera 70 may analyzemovement of the marker through the photo captured by the camera 70 andthen interpret movement of the drawer 50 and the direction of themovement. At this time, the analysis of the marker may be conducted bythe drawer sensing unit 130. That is, the drawer sensing unit 130 maydetermine the state of the drawer 50 by analyzing movement of the markerin the pictures which were continuously taken. Therefore, depending uponthe result of sensing or determination by the drawer sensing unit 130,the time at which a picture corresponding to the third region may bedetermined. The portion corresponding to the third region and the otherportions in the acquired picture may be independently and individuallyselected, separated and updated.

FIG. 32 shows a simplified image of the camera and a wording reading as“SMART SENSOR.” The marker may include a pattern of various shapes. Thepattern may be formed by alternately arranging the white and blackcolors such that the pattern is recognizable by the camera 70.

The marker may be indicated at the upper side of the front surface ofthe drawer 50.

FIG. 33A is a view illustrating a photo showing the left and rightdrawers which are in the withdrawn position, and FIG. 33B is a viewillustrating a photo showing the left drawer which is in the introducedposition and the right drawer which is in the withdrawn position.Hereinafter, a description will be given with reference to FIGS. 33A and33B.

FIGS. 33A and 33B illustrate an example of the marker arranged at thecenter of the upper side of the drawer 50 and provided with a wavepattern.

The first region 42 may be shown in one photo captured by the camera 70.In addition, one photo may show one of the second region 52 and thethird region 62.

That is, a photo captured with the left and right drawers withdrawn asshown in FIG. 33A shows the first regions 42 disposed on the left andright sides and the second regions 52 disposed on the left and rightsides.

On the other hand, a photo captured with only the right drawer withdrawnas shown in FIG. 33B may show the first regions 42 disposed on the leftand right sides, the third region 62 disposed on the left side, and thesecond region 52 disposed on the right sides.

That is, when the drawer 50 is withdrawn, the photo shows the secondregion but does not show the third region 62. On the other hand, whenthe drawer 50 is introduced, the photo shows the third region 62 butdoes not show the second region 52.

The camera 70 captures photos at the same position. The drawer 50 ismovable, and thus the camera 70 may capture a photo of various regionsaccording to the position of the drawer 50.

When the drawer 50 arranged on the left side is fully introduced intothe space under the shelf 40 as shown in FIG. 33B, the marker is notrecognized by the camera 70.

That is, the marker is not shown in a photo captured by the camera 70.

Hereinafter, a method of sensing movement of the drawer using the markerwill be described with reference to FIGS. 34 to 40.

FIG. 34 is a flowchart describing operation of the refrigeratoraccording to one implementation.

Referring to FIG. 34, when the door switch 110 senses opening of thedoor 20, the camera 70 starts capturing a photo (S2101), and thecontroller 100 starts recognizing and tracing the marker (S2103).

Hereinafter, steps S2101 and S2103 will be described in more detail withreference to FIG. 35.

FIG. 35 is a flowchart describing a method of recognizing and tracingthe marker according to one implementation.

Referring to FIG. 35, when the door 20 is opened, the camera 70 startscapturing a photo (S2301).

The controller 100 controls the camera 70 to acquire a photo of theinterior of the drawer 50 (S2303).

Subsequently, the controller 100 processes the acquired photo (S2305),and determines whether a predetermined marker is recognized in theacquired photo (S2307).

To determine whether the predetermined marker is recognized, thecontroller 100 may compare if the marker recognized in the acquiredphoto is identical to the prestored marker.

The predetermined marker may be determined by the provider or the user,and may be pre-stored in the storage unit 18.

A marker may be formed in a searchable region in a photo captured by thecamera. Hereinafter, a detailed description will be given of a markerwhich may be recognized and traced by the controller 100, with referenceto FIGS. 36A through 37B.

FIGS. 36A and 36B are views illustrating a marker for sensing the timewhen the drawer is introduced or withdrawn according to oneimplementation.

Referring to FIG. 36A, the marker may be formed in a searchable regionby the camera 70 from the time the drawer 50 starts to be opened. Forexample, in the case that the xx070 is positioned at the upper end ofthe refrigerator and allowed to capture a plan view of the drawer 50,the marker may be positioned at the front edge portion of the upper endof the drawer 50. In addition, the camera 70 may be installed to capturea photo of the drawer 50 from the position at which the drawer 50 isfully closed, to the position at which the drawer 50 is fully opened.

In the case that the marker is positioned at the upper end of the drawer50, it is presumed that there is no obstacle at the upper end of thedrawer 50 at the time the drawer 50 is closed after items stored in thedrawer 50 are arranged in order. However, implementations are notlimited thereto.

Referring to FIG. 36B, when the xx54 accommodating the drawer 50 and thedrawer 50 are viewed from the point of view of the camera 70, the camera70 may capture a photo of the interior of the drawer 50 including themarker within the range from the position at which the drawer 50 beginsto be opened to the position at which the drawer 50 is fully opened.

FIGS. 37A and 37B are views illustrating the shape of a marker accordingto one implementation.

Referring to FIGS. 37A and 37B, the marker may be formed by alternatelyarranging identical shapes with two different colors

Referring to FIG. 37A, the marker may be formed by arranging a pluralityof bands having white squares and black squares which are alternatelyarranged such that the bands are misaligned.

In the case that the marker has a repeated pattern as above, even when aportion of the marker is not in the capture range of the camera, thecontroller 100 may recognize and trace the marker.

The marker is not limited to a repeated pattern. Nor is it limited to aplurality of colors. The marker may have any pattern as long as itallows a movable internal constituent of the refrigerator such as thedrawer 50 to be recognized.

Referring back to FIG. 35, when the predetermined marker is recognized,the controller 100 senses movement of the recognized marker (X2311). Thecontroller 100 may analyze and process the photos captured consistentlyor periodically, thereby determining whether the recognized marker movesor stops. For example, the controller 100 may determine whether therecognized marker moves or stops in the case that opening of the drawer50 is continued or completed, or in the case that closing of the drawer50 is started, continued, or completed.

Alternatively, in the case that the predetermined marker is notrecognized, the controller 100 informs the user that recognition of themarker has failed (S2309).

For example, in the case that an obstacle is positioned at the marker,the camera may fail to capture a photo including the marker, or thecontroller 100 may not recognize the marker in an acquired photo. Inthis case, the controller 100 may provide the user with informationindicating that recognition of the marker has failed or informationproviding a way to succeed in recognizing the marker such as removingthe obstacle, through the display 14.

Subsequently, in the case that recognition of the marker fails, thecontroller 100 may process a new photo acquired through the camera andrepeat the steps for recognizing the predetermined marker.

Referring back to FIG. 34, when closing of the drawer is sensed (S2105)after the marker is recognized, the controller 100 acquires a finalphoto including the interior of the drawer (the third region 62) througha photo captured at the corresponding time (S2107). At this time, thefinal photo of the interior of the drawer acquired by the controller 100may include a view of the interior of the drawer immediately beforeclosing of the drawer is sensed, or include a view of the interior ofthe drawer immediately after closing of the drawer is sensed. However,implementations are not limited thereto.

Sensing closing of the drawer will be described with reference to FIG.38.

FIG. 38 is a flowchart illustrating a method of sensing closing of thedrawer according to one implementation.

Referring to FIG. 38, the controller 100 determines whether terminationof opening of the drawer 50 has been sensed (S2501). For example, in thecase that the controller 100 recognizes the marker in the previouslyacquired photo of the drawer 50 and senses, by tracing movement of themarker, that the marker has stopped moving, it may sense termination ofopening of the drawer 50.

Subsequently, the controller 100 determines whether start of closing ofthe drawer 50 has been sensed (S2503). For example, in the case that themarker stops moving and moves again in the reverse direction, thecontroller 100 may sense that closing of the drawer 50 has started.

Hereinafter, operation of the controller 100 having sensed closing ofthe drawer according to another implementation will be described withreference to FIG. 39.

FIG. 39 is a flowchart illustrating an operation of the refrigeratorthat is performed when termination of closing of the drawer is sensedaccording to one implementation.

When termination of closing of the drawer is sensed (S2701), thecontroller 100 determines whether acquisition of a final photo throughrecognition of the marker has been successful (S2703).

The controller 100 may sense termination of closing of the drawer 50through image processing by the drawer sensing unit 130.

In the case that the marker is not normally recognized due to, forexample, an obstacle and thus acquisition of the final photo failed, thecontroller 100 informs the user that recognition of the marker hasfailed (S2705).

The controller 100 may provide the user with information indicating thatrecognition of the marker has failed through, for example, the display14. Thereby, it may guide the user in removing the obstacle andintroducing or withdrawing the drawer such that the marker is normallyrecognized and the final image is acquired,

Referring to FIG. 34, the controller 100 displays the final image using,for example, the display 14. At this time, the image provided for theuser may be the image of the portion corresponding to the third region62 extracted from the acquired final photo. Hereinafter, control of afinal image according to another implementation will be described withreference to FIG. 40.

FIG. 40 is a flowchart illustrating a method of controlling an image ofa certain region in the refrigerator captured at a certain timeaccording to another implementation.

Referring to FIG. 40, the controller 100 acquires the final photo(S2901), and then may cut an image of a desired region off of the photo.Then, the acquired final image is stored in the storage unit 18 (S2903).

Subsequently, the controller 100 determines, through user inputs,whether a user input for display of the final image is received (S2905).

For example, when the final photo is acquired, the controller 100 mayprovide information indicating success of acquisition of the final phototo the display 14 constructed with a touchscreen and form an icon typeuser input portion on the display 14. Whether or not a user input isreceived may be determined depending on whether the corresponding iconis selected. The user input portion may not only include the icon formedon the touchscreen but also a physical key described above. However,implementations are not limited thereto.

Subsequently, when a user input for displaying the final image obtainedby processing the final photo is received, the controller 100 displaysthe final image (S2907). The controller 100 may display the final imageusing the display 14. At this time, the displayed existing image may bereplaced with the final image. Accordingly, displaying the final imagemay be viewed as updating of the final image.

Updating the final image may be automatically performed regardless ofthe user input.

According to one implementation, the refrigerator may sense movement ofthe drawer by recognizing a marker pre-formed on, for example, thedrawer. The refrigerator may acquire an image of the final state of anitem stored in the drawer using a camera installed in the refrigerator.At this time, the refrigerator may determine the optimum time at whichthe final storage state of, for example, the drawer, using the markerformed on the drawer.

Hereinafter, a description will be given of a position where the camerais installed and acquires photos according to the position.

According to one implementation, the camera 70 may be installed outsidethe drawer 50, i.e., in the storage compartment 22 and continuouslycapture photos including the interior of the drawer 50 and the movementpath of the drawer body (including the front and rear walls of thedrawer 50 and the upper portion of both sidewalls of the drawer 50′forming the drawer 50. Thereby, not only the state of the items storedin the drawer but also movement of the drawer may be simultaneouslyrecognized. A photo captured by the camera 70 also contains an image ofthe space at the upper side of the drawer 50, i.e., the portioncorresponding to the first region 42.

Basically, a photo captured by the camera needs to include both theoperation of opening and closing of the drawer and the path along whichthe drawer moves. Accordingly, the camera needs to be installed at aposition outside the drawer allowing the camera to capture a photo ofopening/closing operation of the drawer and the movement path of thedrawer. That is, the camera is preferably installed at a position whereit can capture a photo including images of both the closed position andfully opened position of the drawer.

Further, the camera 70 may be configured to cover storage regionsincluding the drawer region, i.e., a plurality of storage regions suchthat a photo captured through one shooting contains not only the drawerregion but also the storage compartment or other storage regions suchas, for example, the shelf region in which food or items divided by theshelf are stored.

To this end, the drawer 50 is preferably formed on the bottom surface inthe storage compartment of the refrigerator. The drawer 50 may include afront wall, two sidewalls, a rear wall, a bottom wall, and an openingformed at the upper portion of the drawer 50. The camera 70 needs to bearranged over the drawer such that at least one portion of the interiorof the drawer 50 and the upper portion of the front wall of the drawer50 are exposed to the camera 70 when the drawer 50 is fully opened.

In addition, a printed part (a marker) with a specific pattern is formedor attached at a position on the front wall or the upper surface of thesidewalls of the drawer exposed to the camera.

At this time, the camera 70 is fixed at a predetermined position, andthus the position of the printed part of the drawer 50 needs to berelatively changed. That is, when the camera 70 receives a drive commandfrom the controller 100, it continuously captures images at a constantcapturing speed, and the captured images of the interior of therefrigerator are sequentially transmitted to the controller 100. Thecontroller 100 traces the position of the printed part based on thephotos of the items stored in the refrigerator received from the camera70. Thereby, it determines information about, for example, a degree ofopening of the drawer and movement of the drawer.

The range of capture by the camera may be determined by an angle of viewof the camera. The angle of view includes a vertical angle of viewrepresenting the vertical range of capture and a horizontal angle ofview determining the horizontal range of capture.

One end of the vertical angle of view of the camera 70 needs to bedetermined such that the front wall of the drawer 50 is within thevertical angle when the drawer 50 is at least fully opened. The otherend of the vertical angle of view needs to be determined such that aportion, e.g., a portion of the front surface of the upper side ofdrawer 50 covering the opening at the upper portion of the drawer 50allowing checking of whether the printed part positioned on the frontwall or upper surface of the sidewalls of the drawer 50, is not exposed,is within the vertical angle when the drawer 50 and is closed.

At this time, it may be possible for one camera to recognize theinternal region of the drawer and another storage space simultaneously.According to another implementation, a separate accommodation space(hereinafter, a bottom accommodation section of the storage compartment)may be provided on the bottom surface of the storage compartment, whichis arranged between the front wall of the drawer in the fully closedposition and the opening in the front surface of the storage compartmentand overlaps at least one portion of the drawer when the drawer isopened. In this case, the camera 70 may be configured to recognize theinternal region of the drawer 50 and another storage space (the bottomaccommodation section) together in the range of capture in which thevertical angle of view of the camera 70 covers the fully opened positionand fully closed position of the drawer.

Meanwhile, in another implementation for the vertical angle of view ofthe camera 70, a photo of both the internal region of the drawer and theshelf region may be captured. Accordingly, the range of the other end ofthe vertical angle of view may be determined such that the verticalangle of view covers at least one part of a plurality of shelves whichare disposed vertically spaced a certain distance from each other.Preferably, in configuring the camera such that the vertical angle ofview thereof covers the shelf region, the vertical angle of view maycover even a portion of the uppermost shelf of the shelves. Morepreferably, the vertical angle of view may cover even the rear edge ofthe uppermost shelf disposed at the uppermost end of the storagecompartment.

According to the illustrated examples, the stored condition of the itemsin the at least two storage spaces including the internal region of thedrawer may be checked through one photo captured by one camera.

To this end, the camera needs to be installed at a desirable positionwith respect to the drawer formed at the bottom portion of the storagecompartment. Inventors of the present invention reviewed positions forinstallation of the camera disclosed in or known from documentspublished at the time of filing of the present application. First, itwas reviewed whether a photo of the interior of the storage compartmentcan be captured by installing the camera at the upper end of the dooroutside the storage compartment of the refrigerator.

In the case that the camera is installed at a position (the door)outside the storage compartment, the camera is kept exposed to theexternal temperature and thus dew condensation does not occur around thecamera. Accordingly, a separate structure to prevent dew condensation isnot needed. However, it is difficult to photograph the interior of thestorage compartment with the camera mounted to the door since the useralways puts items in the interior of the drawer and the storage spacesin the compartment with the door open. Particularly, one object of thepresent invention is to obtain the condition of the items stored in thedrawer as well as movement of the drawer from the outside of the drawer.Since the user's operation in the drawer is always performed with thedoor open, and the drawer is closed when the door is closed, this casedoes not serve the mentioned one object.

Accordingly, to photograph at least the interior of the drawer andmovement of the drawer, it is preferable to dispose the camera in thebody of the storage compartment of the refrigerator in which the drawerto be photographed is present rather than disposing the camera outsidethe refrigerator so that the storage compartment can be captured.

In the case that the camera is installed under one of the shelvespositioned over the drawer and oriented toward the opening of thestorage compartment of the body in accordance with the one object, theshelf region cannot be photographed, and it is difficult to install adrive unit to drive the camera and a wire on the shelves. In addition,to cover the entire movement path of the drawer, a wide angle cameraneeds to be used. Accordingly, captured images may be significantlydistorted.

In the case that the camera is installed at a sidewall of the storagecompartment in a range in which the drawer can be exposed to the camera,it has been found that an asymmetric image is produced due to differentdistances from the camera to the left side and right side of the storagecompartment of the refrigerator, and thus an image showing the conditionof the items stored in the drawer is significantly distorted.Accordingly, the level of user satisfaction with the image may begreatly lowered.

Above all, one object of the present invention is to senseopening/closing of the drawer and the degree thereof by tracing movementof the drawer without using a separate sensor. The aforementionedasymmetric image may show a relatively enlarged image (image distortion)of the printed part (the marker) positioned on the front wall of thedrawer or the upper surface of both sidewalls the drawer may be shown.Accordingly, the size (pixel size) of the portion occupied by the markermay increase relative to the whole size of the image and thus change ofposition of the marker may not be easily recognized. In addition, therange to be processed to determine change of position of the marker mayincrease, and thus images which are continuously transmitted may bemissed or fail to be processed.

Accordingly, it has been found that installing the camera at thesidewall or corner of the storage compartment which is known at the timeof filing of this application does not serve the object of this presentinvention.

In one implementation, the camera is installed at the upper wall of thestorage compartment corresponding to the open upper surface of thedrawer within the ranger of the upper surface onto which the drawer inthe closed position and the drawer in the opened position are projected.In addition, the camera is oriented toward the rear wall of the storagecompartment to face the interior of the storage compartment and isinclined at a certain angle such that the focus of the camera isdirected to one point on the rear wall surface of the storagecompartment. Thereby, images undergoing minimized distortion may beobtained with respect to the vertical angle of view of the camera in theillustrated examples.

According to anther implementation, in determining the position of thecamera, the camera is installed at a position on the upper wall of thestorage compartment between the opening in the front surface of thestorage compartment and the projected position on the upper wall of thefront edge portion of the uppermost one of the shelves verticallydisposed in the storage compartment. In addition, the camera is orientedtoward the rear wall of the storage compartment to face the interior ofthe storage compartment and is inclined at a certain angle such that thefocus of the camera is directed to one point on the rear wall surface ofthe storage compartment.

The camera is preferably positioned at the central portion of the upperportion of the storage compartment, and the printed part correspondinglypositioned on the upper surface of the front wall or both sidewalls ofthe drawer is preferably installed on the upper surface of the frontwall of the drawer. More preferably, the position of the camerainstalled on the upper surface of the storage compartment is nearby theupper surface of the front wall of the drawer under the upper surface ofthe storage compartment.

In a refrigerator provided with at least one door, i.e., the left andright doors to open and close the opening of a storage compartment, thecamera is installed on a portion of the upper surface (ceiling surface)of the storage compartment on the boundary between the left and rightdoors, and the printed part is positioned, when left and right drawersare provided, near the boundary between the left and right drawers. Thatis, when a viewer faces the interior of the storage compartment, theleft drawer is preferably positioned at an end of the upper surface ofthe right sidewall or the upper right surface of the front wall, and theright drawer is preferably positioned at an end of the upper surface ofthe left sidewall or the upper left surface of the front wall. In otherwords, printed parts are preferably formed near a position right underthe camera. This is because distortion of the printed parts may occur ifthe printed parts are far apart to the left and right sides from theposition right under the camera. This example is shown in FIG. 41B.

In the case that the printed parts are installed at one side of theupper surface of each of the left and right drawers or an end of theupper surface of each of both sidewalls of the left and right drawers,the drawers may be used in common regardless of the left and rightpositions thereof when they are used for a refrigerator which is openedand closed by the left and right doors. Accordingly, the efficiency ofuse of the drawers may increase.

According to the positions of the camera and printed parts as describedabove according to some implementations, the fixed range of capture ofthe camera covers the movement path of the drawer from the closedposition to the open position. Therefore, not only opening/closing ofthe drawer but also the degree of opening/closing and stopping or movingof the drawer may be sensed and at the same time even the condition ofthe items stored in the drawer may be sensed or checked with one camera.

In addition, the condition of the items stored in the bottomaccommodation section installed on the bottom surface of the storagecompartment may also be checked with the drawer closed. Therefore, thecamera may recognize a plurality of storage regions.

In the case that the angle of view of the camera is extended up to theshelf region, the camera may selectively recognize three storage regionsincluding the internal region of the drawer, the bottom accommodationsection of the storage compartment, and the shelf region.

Since the camera installed on the upper surface of the storagecompartment and the printed part formed on or attached to the upperportion of the drawer are approximately vertically aligned, the image ofthe printed part in a whole image of the interior of the refrigeratorcaptured and transmitted by the camera is less distorted. In addition,the space occupied by the printed part may be relatively minimized, andtherefore the time taken to trace and determine the position of theprinted part through image processing may be minimized.

Hereinafter, a method of tracing and recognizing a marker formed on thedrawer will be described.

A printed part (marker or printing paper) which is recognizable throughimage interpretation may be installed at a specific portion of the bodyof the drawer exposed to the camera. By tracing the position of themarker, the degree of opening/closing of the drawer and the closed oropened state may be sensed. By allowing the marker on the movement pathof the drawer from the closed position to the fully opened position tobe exposed to the camera, the opened or closed state of the drawer maybe determined based on the position of the marker. That is, informationabout various states of the drawer may be determined based on the changein position of the marker in the photos which have been continuouslycaptured.

The printed part may be provided with a specific pattern to trace andrecognize movement of the drawer or the opened or closed state of thedrawer. In this case, the printed part may be disposed at variouspositions on the upper surface of the drawer exposed to the camera.

Preferably, the printed part is installed approximately on a boundary ofthe maximum access opening of the drawer at the upper end of the frontwall or both sidewalls of the drawer.

The pattern of the printed part may be formed in a manner that figureshaving a certain shape and contrasting colors are alternated. Thecertain shape may include quadrangle, triangle, circle, and wave.

The printed part may degrade the whole aesthetics of the refrigeratorand provoke negative feeling among some users. Therefore, repetition ofthe figures of the pattern need to be minimized so as not to providenegative feeling regarding aesthetics of the refrigerator, and a printedpart having a shape attractive to the user needs to be considered.

However, this design-related issue is not simply raised in the aspect ofdesign. It may also be raised in the case that the position of thedrawer is not accurately recognized. That is, the design issue mayaffect the rate of recognition of the printed part of the drawer,thereby resulting in failing to accurately sense the position of thedrawer.

That is, considering that the size of the printed part needs to begreatly reduced in the design aspect, a pattern in which figures havinga certain shape and contrasting colors are repeated two to four timesrather than being repeated more times may be selected. In this case, thecontroller configured to process captured images of the interior of therefrigerator may not accurately recognize the boundary of the printedpart (marker) in a whole image.

According to one implementation addressing this problem, colorscontrasting that of the drawer, which is usually bright-colored,transparent, or semi-transparent, may be selected. In addition,referring to FIG. 41A, the length L1 of the printed part in thedirection perpendicular to the direction of movement (back and forthmovement) of the drawer may be set to be longer than the length L2 ofthe printed part in the direction of movement of the drawer. In theprinted part, a first color portion having a certain length in thevertical direction is formed, and a second color portion having a colorcontrasting the color of the first color portion or identical to thecolor of the drawer and a width approximately equal to that of the firstcolor portion is formed at the front or back of the first color portionwith respect to the movement direction of the drawer. Subsequently, thefirst color portion is repeated. That is, the first color portion andthe second color portion may be sequentially disposed. Regarding to thewhole size of the marker including a first color portion, a second colorportion and anther first color portion disposed in this order, thelength L1 of the marker in the direction perpendicular to the directionof movement of the drawer is preferably longer than the length L2 of themarker in the movement direction of the drawer.

According to another implementation, both ends of the first colorportion arranged at the front edge of the marker are connected to bothends of the first color portion arranged at the rear edge of the markersuch that the second color portion is surrounded by the first colorportions. The marker in the previously disclosed third implementationmay be an example having this structure.

That is, in a preferred implementation of the marker proposed by thepresent invention, the marker is configured such that the length L1 inthe direction perpendicular to the movement direction of the drawer isgreater than the length L2 of the movement direction of the drawer andthat a portion of the marker adjoining the drawer (with respect to themovement direction of the drawer) has a color (a dark color) contrastingthe color of the drawer, and the central portion of the marker has abright color contrasting the dark color.

When the printed part is configured as above, the printed part may havea small size. Accordingly, the user's demand for aesthetics may be metand an image portion of the printed part in a whole image captured withthe camera may have a small size relative to the size of the wholeimage. Accordingly, the size of an area undergoing image processing tosense change in position of the printed part may be reduced and thus theimage processing speed may be improved. This advantageously allows theimage processing speed of the controller to support the speed ofcontinuous capturing of photos to be transmitted. Thereby, the accuracyof acquiring a photo at the accurate time may be further improved. Forexample, if a portion in the photos which are continuously captured islarge, the controller may be subject to a heavy work-load. This may leadto use of a highly expensive controller. If the portion to be processedis small, on the other hand, the load to the controller may be lessened.Accordingly, images may be effectively processed even with aninexpensive controller.

Preferably, as shown in FIG. 41B, the marker is provided with a letter,a sign, or a pattern to allow the user to intuitively understand the useof the marker.

Hereinafter, a description will be given of a method of recognizing thecoordinate of the marker and processing photos from the camera.

FIG. 42 is a view illustrating a method of recognizing the position ofthe marker. Hereinafter, a method of recognizing the position of themarker of the drawer will be described with reference to FIG. 42.

When the user desires to newly introduce (stock) or retrieve an iteminto or from the drawer, the user first opens the door. Once the door isseparated from the front surface of the storage compartment, the doorswitch senses the separation, and the controller receives a signal fromthe door switch and drives the camera which is installed in therefrigerator and set in the standby mode. When the camera receives adrive command from the controller, a drive voltage is applied to thecamera in place of the standby voltage. Then, the camera finishespreparation for driving and starts continuous photographing.

The user will subsequently grip a handle on the front wall of the drawermounted in the storage compartment of the refrigerator and pull thedrawer out to open the drawer. As the camera continuously photographsthe drawer, the camera captures photographs of the opening of the drawerat certain intervals and sends the captured photos to the controller.

Meanwhile, a marker having a certain shape is formed at a portion of thedrawer exposed to the camera. Upon receiving photos sent from thecamera, the controller divides a whole image by a certain set of pixelsand assigns coordinate values to the image at certain intervals. Then,the controller analyzes the coordinate of the marker on the coordinatesof the image.

Accordingly, when a photo captured with the drawer in the closedposition is sent to the controller, the marker will not be exposed inthe corresponding image. At this time, the coordinate value of themarker is 0.

When the user opens the closed drawer, the camera photographs the drawerwith the marker exposed. Thus, the camera captures a photo including themarker. When this photo is sent to the controller, the controllerrecognizes a coordinate value of the marker in the corresponding imageand compares the coordinate value with that of the previous position ofthe marker. For example, referring to FIG. 42, when the coordinate valueof the marker changes from 0 to, e.g., 50 at location A, the controllermay determine that the drawer is moving, based on the change of thecoordinate value. Particularly, when the coordinate value is justchanged from 0, the controller may determine that the drawer has justbegun to open.

When the user continues to open the drawer, the controller maydetermine, through a series of processes as described above, that thedrawer is moving to be opened based on the change of the coordinate ofthe marker to a greater value.

When the user fully opens the drawer, the controller recognizes that thecoordinate value does not change any more and determines that the draweris fully open or that the drawer stops opening in the open state. Forexample, in FIG. 42, location B is a point at which the coordinate valueof the marker is 90 and does not change, and thus the controller mayrecognize that the drawer is stopped or determine that the drawer isfully open.

Normally, when the drawer is opened, whether fully opened or partiallyopened, and then stopped, it may be determined that the user isperforming a certain operation for the drawer. Even when the drawer isstopped, the camera continues to capture photos of the interior of therefrigerator and send the captured photos to the controller. And thecontroller monitors change of the coordinate of the marker.

Once the user's operation of stocking or retrieving an item in or fromthe drawer is completed, the user closes the drawer. At this time, thecontroller recognizes based on the captured photos that the coordinateof the marker in a whole image changes and determines that the drawer ismoving again. For example, in FIG. 42, location C is a point at whichthe coordinate value of the marker changes from 90 to 70. Based onchange of the coordinate value to a lower value, the controller maydetermine that the drawer is beginning to be closed immediately afterthe user completes an operation for the drawer.

Similarly, in the case that the coordinate value of the marker is 0,that is, the marker is not recognized in an image, it is determined thatthe drawer is fully closed.

A description has been given above of a method for the controller torecognize the position of the marker through the coordinate of themarker on a whole photo captured by the camera and to trace the markerand determine whether the drawer is moving or stopped, the degree ofopening of the drawer, and whether the drawer begins to be opened orclosed.

Hereinafter, a description will be given of a method of processingphotos from the camera and acquiring a desired photo based on theposition of the marker of the drawer. Based on the movement state of thedrawer, a method of processing and acquiring a photo of the interior ofthe drawer, particularly a method of obtaining a photo determined to becaptured immediately after the user completes an operation for thedrawer will be described in detail.

When the coordinate of the marker of the drawer changes from 0 to agreater value and thus the drawer is determined to be moving in theopening direction, the controller only senses change of the coordinatebased on a photo of the interior of the refrigerator captured by thecamera and sent to the controller and buffers the whole photo in atemporary buffer without processing the storage regions separately shownin the photo.

When the drawer continues to be opened set in the fully open state, oris partially opened and stopped, the user performs an operation for thedrawer. This situation is determined to occur by the controller whenthere is no change in the coordinate of the marker. Then, a sub-imagecorresponding to the drawer region among the divided regions in a wholeimage of the transmitted photo is moved to and stored in a temporarydrawer buffer. That is, referring to FIG. 42, in a section correspondingto location D, the controller cuts off a necessary part from a portionfor the drawer region of a photo of the entire interior of therefrigerator and moves the part to a temporary buffer for the drawerregion. This section is defined as a “section of refreshing the photo ofthe internal region of the drawer.”

In other words, images are processed differently between the section inwhich the drawer is being opened and the section in which the drawer isstopped. This is because image processing is conducted in considerationof the fact that the user's operation for the drawer is performed withthe drawer stopped. Thereby, the speed of image processing for tracingthe change of the coordinate may increase in the section in which thedrawer moves, ensuring efficient data processing.

When the drawer in the stopped state begins to be closed, the controllerdetermines, based on the change of the coordinate to a lower value, thatthe user's operation has been completed. At this time, the controllerselects and stores the latest image of the images temporarily bufferedin the temporary buffer for photos of the interior of the drawer. Thatis, an image of the internal region of the drawer provided immediatelyafter completion of the user's operation is captured.

The captured (stored) latest image of the items stored in the drawerregion replaces an image on an area of the display, which is connectedonline or directly to the controller, or is transmitted to a networkserver. The latest image transmitted to and stored in the network servermay be provided for the user such that the user checks the image throughthe display installed at the refrigerator or a mobile terminal connectedto the network server whenever the user desires to check.

Next, a description will be given below of a method of selecting andsearching a specific section for the position of the marker rather thansearching a whole captured photo showing the drawer.

That is, it is possible to set a search range for the drawer region. Themovement path of the drawer may be divided into several sectionsaccording to the degree of opening of the drawer and the marker may betraced with different methods in the respective sections. Thereby, timetaken to process (analyze) photos captured and transmitted by the cameramay be shortened such that the photo processing speed matches with thephotographing speed of the camera.

Generally, when the drawer used in the refrigerator is in the full openposition, it is not withdrawn to be separated from the storagecompartment. That is, the normal withdrawn distance of the drawer,commonly referred to as an open distance, is generally about 50% of theentire depth of the drawer (the front-to-back length of the drawer withrespect to the movement direction of the drawer).

The withdrawn distance of the drawer may be determined to be a region inwhich a coordinate is assigned to the marker of the drawer in a showimage to be processed by the controller. In this case, the controllermay recognize the position of the drawer from the time at which thedrawer begins to be withdrawn. Particularly, the withdrawn distancecorresponds to the size of data to be processed by the controller tosense change of the coordinate of the drawer. In addition, to sense anddetermine the change of the coordinate, the controller needs to comparetwo photos, and the speed of performing this process should correspondto the capturing speed of the camera. Preferably, the capturing speed ofthe camera (the time interval at which a photo is captured) isdetermined such that it approximately matches with the speed of thecontroller determining the change based on the transmitted photos andprocessing the photos.

According to this method, the marker is immediately located in thephotos captured by the camera, and thus the position of the drawer isdetermined. In addition, the controller may easily obtains images fromnew photos continuously sent from the camera, while processing dividedregional images of the storage spaces in a whole photo of the interiorof the refrigerator. That is, depending upon whether the door is openand whether the drawer is open, images of the respective regions may becut off from the whole image and stored in the corresponding temporarybuffers, without leaving any photo captured by the camera unprocessed.

However, data for two images (a previous image and a current image)having the size as described above need to be processed, and accordinglya high processing capability of the controller may be required. In thecase that the processing capacity of the controller is increased to meetthis requirement, the price of a processor may rise significantly. Inaddition, to process images without the drawer open, an excessively highprocessing capability may be required and thus efficient selection ofcomponents may not be implemented.

According to one implementation provided to address this problem, imagesidentifying the drawer identifying the drawer region are separated stepby step and the image data are processed with different methodsaccording to the withdrawn distance, considering that it issubstantially difficult to sufficiently recognize the interior of thedrawer with the drawer open by a certain distance.

FIGS. 43A to 43C are views illustrating the degree of opening of thedrawer. FIG. 43A shows the drawer which is fully open, i.e., open byabout 50% of the overall length of the drawer, FIG. 43B shows the drawerwhich is open by about 30% of the overall length of the drawer, FIG. 43Cshows the drawer which is open by about 10% of the overall length of thedrawer.

For example, when the user opens the drawer by about 10% of the overallfront-to-back length of the drawer as shown in FIG. 43A, the itemsstored in the drawer may be not sufficiently identified. Only when thedrawer is opened by about 30% of the overall front-to-back length, theitems stored in the drawer may be identified to a certain extent, andthus the drawer may be recognized as being open.

When the drawer is fully open as shown in FIG. 43A, the drawer has beenwithdrawn by about 50% of the overall length of the drawer, and theitems stored in the drawer may be sufficiently identified. In otherwords, only when the drawer is open by about 30% to about 50% of theoverall length, the user can store or retrieve an item in or from thedrawer. When the drawer is in this section of movement it may be viewedas being open.

In this context, when the withdrawn distance of the drawer is less than30% of the overall length, it may be determined that “the drawer is notopen.” When the is withdrawn distance of the drawer equal to or greaterthan 30% of the overall length, it may be determined that “the drawer isopen.” Thereby, photos may be processed differently according to thewithdrawn distance to reduce load of data that the controller need toprocess.

The controller obtains images from photos captured by the camera anddetermines the image size corresponding to the movement trajectory ofthe drawer. Then, the controller determines the number of pixels to thepoint or distance at which the drawer is withdrawn by about 30% of theoverall length and sets an image section.

That is, when the drawer is withdrawn from the position shown in FIG.43A to the position shown in FIG. 43B, the controller may determinechange of the coordinate of the marker. In this case, load to the cantroller may be reduced compared to the case in which change of thecoordinate is determined when the drawer is withdrawn from the positionof FIG. 43A to the position of FIG. 43C.

In the movement section from the closed position of the drawer to theposition where the drawer is withdrawn by 30% of the overall length,only whether the marker has reached the position of a pixelcorresponding to the distance of 30% of the overall length isdetermined. In other words, the coordinate of the marker in the previousphoto is not compared with the coordinate of the marker in the currentphoto (to determine change of the coordinate of the marker). Rather,only whether the position of the marker is within a certain pixel rangeis determined and the whole image is temporarily buffered.

Since only the data of one photo is processed in this movement sectionwithout comparing two photo's data, the capability of the controller toprocess data does not need to be increased, the processing may besmoothly performed according to the capturing speed of the camera.

Meanwhile, in the case that the drawer is opened beyond the positioncorresponding to 30% of the overall length, that is, from the time atwhich the marker reaches a position beyond a specific pixel in a photoprocessed by the controller, the value of the coordinate of the markermay be determined by comparing two photos (the last photo capturedimmediately before this time and a photo captured after this time) todetermine whether the drawer is stopped.

If the search range is limited as above, the size of a photo is reducedto about ⅖ of the size of the photo obtained when the search range isnot limited. Further, the photo size corresponding to two photos to beprocessed is reduced to about ⅘ of the size of the photo obtained whenthe search range is not limited. Accordingly, the size of the data to beprocessed by the controller may become substantially less than or equalto the size of one whole photo.

Accordingly, the speed of data processing for tracing the change of thecoordinate value between two photos may correspond to the capturingspeed of the camera, and thus data processing (tracing or monitoring thecoordinate) may be possible without leaving any photo captured by thecamera unprocessed. Time delay related to the processing time may beprevented even when the capacity of the controller is not increased.Therefore, a desired photo may be acquired at the accurate time.

To sum up, it is possible to divide a portion of the distance by whichthe drawer is withdrawable into a first section and a second section ina photo captured by the camera.

The position of the marker may be determined based on the pixels of aphoto captured in the first section. At this time, in the first section,the pixel position of the marker in the photo may be determined withoutcomparing values of the coordinate of the marker.

In the first section, it is difficult for the user to retrieve or storean item from or in the drawer even if the drawer is actually open.Therefore, the drawer may be viewed as being substantially closed.

The second section may be set as a coordinate search range in which thecoordinate values of the marker in two continuously captured photos arecompared. Since two photos are compared in the second section, change ofthe coordinate of the marker may be sensed. For example, if the markeris shown at the same position in the two photos, it may be determinedthat the drawer remained stopped while the two photos are captured. Ifthe position of the marker is changed between the two continuous photosit may be determined that the drawer moved while the two photos arecaptured. At this time, depending upon the direction in which theposition of the marker was changed, it may be determined whether thedrawer is moving to be closed or to be opened.

In the second section, an opening allowing the user to store or retrievean item in or from the drawer is exposed, and accordingly the drawer maybe viewed as being substantially open.

FIG. 45 is a control flowchart according to one implementation.

Hereinafter, a description will be given with reference to FIG. 45.

It is determined that the door 20 opens the storage compartment 22(S10). At this time, opening/closing of the door 20 may be sensed by thedoor switch 110.

When the door 20 is opened, the door sensor 120 may be driven. That is,the door sensor 120 may sense whether the door 20 has rotated an anglegreater than or equal to a certain angle θ. That is, the door sensor 120may not operate when the door 20 is not open, and may begin to operatewhen the door 20 is opened.

In addition, when the door 20 is opened, the camera 70 may begin tocapture photos (S14). At this time, the camera 70 may capture photos atpredetermined time intervals.

For example, it may capture ten photos per second.

The photos captured by the camera 70 are stored in the storage unit 18.In the case that a plurality of photos are captured by the camera 70, amost recently captured photo may be stored and a previously capturedphoto may be deleted due to the capacity limit of the storage unit 18.

Meanwhile, the drawer sensing unit 130 may sense whether the drawer 50is open (S20). The drawer sensing unit 130 may sense withdrawal of thedrawer 50 from the space under the shelf 40.

In addition, when closing of the drawer 50 is sensed by the drawersensing unit 130, a photo captured at this time or at a time closest tothis time is selected as a final photo (with the drawer in the openstate).

Since the camera 70 captures a determined number of photos per second, aphoto may not be correctly captured at the time when the drawer 50begins to be closed. When movement of the drawer 50 is determined byanalyzing, through the drawer sensing unit 130, movement of the markerexpressed on the photos captured by the camera 70, a photo may beselected at the time at which the drawer 50 is determined to begin to beclosed.

The controller 100 divides the final photo into the first region 42, thesecond region 52, and the third region 62 (S42). At this time, thecontroller 100 may divide the photo into the regions by specifying thenumber of pixels from the adjustment line 15 and acquire final images ofthe respective regions.

As described above, when the drawer 50 is withdrawn, the second region52 overlaps the third region 62.

Since information indicating that the drawer 50 has been withdrawn hasbeen obtained in S20, the controller 100 may determine that a secondimage has been obtained.

Then, information about the second region 52 may be updated (S46). Thecontroller 100 may transmit the second image showing the second region52 to the display 14 to change a corresponding portion on the display 14based on the final photo.

On the other hand, when it is determined in S30 that the drawer 50 doesnot begin to be closed, photos may be captured as in S14.

When it is determined in S30 that the drawer 50 is not closed, it maymean that the use has not terminated access to the second region 52.That is, the user may store a new food item in the second region 52 orretrieve a stored item from the second region 52, with the drawer 50stopped.

On the other hand, when it is determined that the drawer 50 begins to beclosed, it may suggest that the user terminates access to the secondregion 52 and introduce the drawer 50 into the space under the shelf 40.

Meanwhile, in the case that it is determined in S20 that the drawer 50is not open, it may be determined whether the door 20 is about to beclosed (S60).

At this time, whether the door 20 is about to be closed may be sensed bythe door sensor 120.

Upon terminating access to the storage compartment 22, the user closesthe door 20, thereby shutting the storage compartment 22 tightly. Thatis, the user may store a new food item in the xx2 or retrieve a storedfood item from the storage compartment 22, with the display 14 opened bythe door 20.

In the case that the door 20 is closed by the user, it may be determinedthat the user terminates access to the storage compartment 22 and thatthe food in the storage compartment 22 is maintained in place until thedoor 20 is opened again.

When it is determined in S60 that the door 20 is about to be closed, thecontroller 100 may select a photo captured at this time or at a timeclosest to this time as a final photo (with the drawer in the closedstate) (S70). That is, the controller 100 may select, as a final photo,a photo captured at the time at which capturing photos with the camera70 is terminated or at a time close to this time.

The controller 100 divides the final image into the first region 42, thesecond region 52 and the third region 62 (S72). At this time, the finalphoto may include a first image showing the first region 42. The mostrecent state of food stored in the first region 42 is the condition ofthe food at the time when the user terminates access to the storagecompartment 22. Accordingly, the user may acquire correct informationabout the food in the first region 42 from the first image acquired atthis time.

Since information indicating that the drawer 50 has been introduced hasbeen obtained in S60, it may be determined that the controller hasobtained the first image and the third image. As described above, whenthe drawer 50 is withdrawn, the second region 52 and the third region 62overlap each other, but the first region and the third region arepositioned not to overlap each other but to be photographed together.

In addition, information about the first region 42 and the third region62 may be updated (S76). The controller 100 transmits the first imageshowing the first region 42 and the third image showing the third region62 to the display 14 such that corresponding portions on the display 14are changed based on the final photo.

Although not shown in FIG. 45, when the door 20 rotates to shut thestorage compartment 22 and thus the storage compartment 22 is closed,driving of the camera 70 may be stopped. At this time, the door switch110 may sense whether the door 20 shuts the storage compartment 22.

That is, in this example, a photo acquired at the time at which the userterminates use of the storage compartment 22 is processed to provide theuser with information about the food stored in a region. Accordingly,the user may obtain accurate information about the food stored in thestorage compartment 22.

Meanwhile, two door sensor 120, two 110 and two drawer sensing unit 130may be installed on the left side and the right side, respectively. Inthis case, individual information about the left and right doors and theleft and right drawers may be acquired. In the case that the drawersensing unit 130 senses movement of the drawer by analyzing the markershown in the captured photos, the marker may be installed at each of thetwo drawers while one drawer sensing unit 130 is provided.

Accordingly, in the case that only the left door is open, only the leftportion of a photo may be updated based on the aforementionedinformation. In the case that only the right door is open, on the otherhand, only the right portion of the photo may be updated.

In the case that both the left door and the right door are open underconditions as above, both left and right portions of a photo may beupdated and provided to the display 14.

FIG. 46 is a control flowchart illustrating a variation of the exampleillustrated in FIG. 45.

Referring to FIG. 46, S60 of FIG. 45 is divided into S62 and S64. Onlydetails different from those of FIG. 45 will be described.

That is, determining whether the door 20 is about to be closed isconducted by determining whether the door 20 rotates in the closingdirection (S62) and determining whether the door 20 rotates an anglegreater than or equal to a certain angle θ.

In other words, when the door 20 rotates in the closing direction (S60),and an angle the door 20 rotates is less than or equal to a certainangle (S64), capturing photos with the camera 70 may be terminated.

To obtain information about food stored in the storage compartment 22from the photos captured by the camera 70, interference with the door 20and a basket installed at the door 20 should not occur. Accordingly, inthe case that the door 20 is about to be closed, it may be suggestedthat the door 20 and other relevant constituents are not shown in thephotos captured by the camera 70.

FIG. 47 is a control flowchart illustrating a variation of the exampleof FIG. 46. Hereinafter, a description will be given of some detailsdifferent from those of FIG. 46 with reference to FIG. 47.

Referring to FIG. 47, the direction of rotation of the door 20 is notdetermined unlike the example of FIG. 46.

When it is determined in S12 that the door has rotated an angle greaterthan or equal to a certain angle, it suggests that the door 20 rotatesin the opening direction at that time. Accordingly, when the userrotates the door 20 again, and thus the door 20 reaches a certainangular position, it may mean that the user is closing the door 20.

Accordingly, in this example, step S62 of FIG. 46 is not performed. Onceit is determined in S20 that the drawer is not open, step S64 ofdetermining whether the door has rotated an angle less than or equal toa certain angle is immediately performed.

FIG. 48 is a control flowchart illustrating another variation of theexample of FIG. 45. Hereinafter, a description will be given withreference to FIG. 48.

FIG. 48 describes a control flow of sensing movement of the drawer 50through the marker. For simplicity of description, details same as thosediscussed above will not be described, and only features different fromthose of the example of FIG. 45 will be described.

S20 and S30 of the previous example may be divided into S22, S24 andS26.

After photos are stored in the storage unit 18 as in S16, the camera 70determines whether the marker is recognized (S22).

Since the camera 70 captures photos, facing downward from the upper sideof the inner case 12, the marker may not be exposed to the camera 70.That is, in the case that the handle of the drawer 50 is not shown atthe front edge of the shelf 40, i.e. in the case that the drawer 50 isnot withdrawn, the marker may not be recognized.

In the case that the marker is recognized in a photo captured by thecamera 70, it may be determined whether the marker is stopped (S24). Atthis time, the movement path of the marker may be identified in onephoto. Then, the path may be divided into the first section and thesecond section and movement of the marker may be determined only in thesecond section.

Since the camera 70 captures a plurality of photos at predetermined timeintervals, whether the marker moves may be determined by comparing thephotos. Since the marker is marked on the drawer 50, the drawer 50 maybe determined to move when the marker moves.

In addition, it may be determined whether the marker has moved adistance greater than or equal to a certain distance from the positionat which the drawer 50 is closed (S26).

In the case that the marker has not moved a distance greater than orequal to a certain distance, the drawer 50 is not sufficiently open, andthus a proper image of the second region 52 may not be acquired. Forexample, in the case that a photo is captured with the drawer 50withdrawn by only ⅓ of the overall length of the drawer 50, the innerspace of the drawer 50 may not be sufficiently shown in the photo, andthe user may not acquire information about the second region 52 from animage acquired from this photo. Herein, the certain distance may bedetermined based on whether the marker has entered a search range fordetermination of movement of the marker.

When it is determined that the marker has moved a distance greater thanor equal is to the certain distance, the determination may mean that theuser has terminated access to the second region 52 and finished use ofthe second region 52.

Accordingly, a photo acquired at the time at which the aforementionedthree conditions are met or a time close to this time may includeinformation about the food stored in the second region 52.

FIG. 49 is a view illustrating another variation of the example of FIG.45.

Hereinafter, a description will be given with reference to FIG. 49.

Unlike FIG. 45, the camera of the example of FIG. 49 is not drivenimmediately at the time when the door switch 110 determines that thedoor 20 is open.

When the door switch 110 determines that the door 20 is open, the doorsensor 120 is driven,

In addition, the door sensor 120 determines whether the door 20 hasrotated an angle greater than or equal to a certain angle (S12). In thecase that the door 20 has not rotated an angle greater than or equal tothe certain angle, the camera 70 does not capture photos.

When the door 20 rotates an angle greater than or equal to the certainangle, the camera 70 captures photos (S14). At this time, the camera 70captures photos at predetermined time intervals. For example, the camera70 captures ten photos per second.

A captured photo includes the first image of the first region 42. Sincethe door 20 has rotated an angle greater than or equal to a certainangle θ, the basket installed at the door 20 may not interfere withcapturing a photo. The basket installed at the door 20 may not appear inthe photo.

In S60, the door sensor 120 may sense that the door 20 is about to beclosed. When it is said that the door 20 is about to be closed, it maymean that the door 20 rotates in the closing direction, and thus thedoor 20 forms an angle less than or equal to a certain angle.

The camera 70 captures photos and stops the capturing operation when thedoor 20 is about to be closed (S69). In FIG. 45, the camera 70 capturesphotos until the door switch 110 is pressed by the closing door 20. Incontrast, in the example of FIG. 49, the camera 70 stops capturingphotos when the door sensor 120 determines that the door sensor 120 hasrotated to a certain angle. The certain angle may be an angle at whichobstruction of the view by the door 20 and the basket installed at thedoor 20 does not occur in the photo.

FIG. 50 is a control flowchart illustrating another implementation.Hereinafter, a description will be given with reference to FIG. 50. Inthe control flow of FIG. 50, when it is determined through the doorswitch 110 that the door 20 is open, the capturing operation may becontinuously performed at predetermined time intervals.

First, whether the door 20 is open by an angle greater than or equal toa certain angle θ is determined by the door switch 110 (S100).

When the door 20 is open by an angle greater than or equal to thecertain angle, it is determined that the capturing operation is notinterfered with by the drawer 50, and the camera 70 captures photos(S110). At this time, the camera 70 may capture one photo.

Then, it is determined whether the drawer has been withdrawn by adistance greater than or equal to a first set distance (S120). Herein,the set distance may be a distance at which food information about thesecond region 52 is sufficiently acquirable from the photos captured bythe camera 70. For example, the first set distance may be ⅔ or ⅓ of theoverall length of the drawer 50.

The withdrawn distance of the drawer 50 may be sensed by the drawersensing unit 130.

When it is determined that the drawer 50 has been withdrawn by adistance greater than or equal to the first set distance, the controller100 divides the photo into a plurality of images according to therespective regions (S122).

Since the photo is acquired with the drawer 50 withdrawn, the firstimage and the second image may be acquired from the photo.

The controller 100 may update the first image and the second image(S124).

On the other hand, when it is determined in S120 that the drawer 50 hasnot been withdrawn by a distance greater than or equal to the first setdistance, it may be determined whether the drawer 50 has been withdrawnby a distance less than or equal to a second set distance (S130).

At this time, the second set distance may be a distance at which foodinformation about the third region 62 is sufficiently acquirable fromthe photos captured by the camera 70. For example, the second setdistance may be ⅓ of the overall length of the drawer 50. The second setdistance may be set to 0 to select the position of the drawer 50 fullyintroduced into the space under the shelf as the second set distance.

When it is determined that the drawer 50 has been withdrawn by thesecond set distance, the controller 100 may divide the photo into aplurality of images according to respective regions (S132).

Since the photo is acquired with the drawer 50 withdrawn, the firstimage and the third image may be acquired from the photo.

The controller 100 may update the first image and the third image(S134).

According to the example of FIG. 50, the control operation is repeatedwhen the door 20 is opened, and is completely terminated when the door20 shuts the storage compartment 22. When the control operation isterminated, image updating is not performed anymore and thus a finalimage of the food stored in the storage compartment 22 may be acquired.

FIG. 51 is a view illustrating a process of updating images with twodoors for opening and closing the storage compartment and two drawersprovided. Hereinafter, a description will be given with reference toFIG. 51.

In describing this example, some of the details identical to those ofthe previous example will be briefly described.

When electricity is supplied to the refrigerator, the door switches 110installed at both left and right sides of the refrigerator mayindividually determine whether the left door or the right door is open(S302 and S303).

When one of the door switches 110 determines the left door or the rightdoor is opened (S303 the camera 70 begins to capture photos (S304).

The camera 70 may capture photos when one of the left door and the rightdoor is opened or both the left door and the right are openedsimultaneously or at different times. This is because light is turned onin the refrigerator when it is determined through one of the doorswitches that a door is opened.

When the drawer disposed on the left side is withdrawn by a distancegreater than 130 mm (S310), an image related to the left drawer iscaptured (S312). The image related to the left drawer may represent theleft drawer shown in the captured photo or may represent the wholecaptured photo.

At this time, the length 130 mm, which may be changed by themanufacturer of the refrigerator or the user, may represent a distanceby which the drawer is withdrawn by the user such that the storage spacein the drawer is accessible.

When the left drawer is withdrawn by a distance less than or equal to130 mm (S310), the last image captured by the camera 70 is maintained(S310).

In addition, the left door sensor senses the angle of opening of theleft door. At this time, in the case that the left door is opened by anangle less than 80 degrees, an image of the left multi-accommodationspace provided in the space under the left drawer in the withdrawnposition (S332) is provided/captured.

At this time, 80 degrees may represent an angle at which the left doorand several structures installed at the left door do not interfere withthe camera 70 when the camera 70 photographs the accommodation space.Depending upon the structure or size of the refrigerator, the angle maychange.

When the left door is opened by an angle greater than 80 degrees, thelast photo captured by the camera 70 may be maintained (S336).

In addition, the door switch 110 may sense that the left door is closed(S334).

For the left drawer and left door, the same control flow as describedabove with respect to the left drawer and left door may be implemented.That is, the descriptions given above is equally applicable to stepsS320 to S324 and steps S340 to S346, and thus the steps will not bedescribed in detail.

When the right or left door is closed, a captured image may be processedby the controller 100 and the processed image may be displayed on thedisplay 14 (S350).

Herein, the display may be a constituent installed at the refrigerator.

Then, the camera may stop the photographing operation (S352).

Meanwhile, it is determined whether the refrigerator is connected withWi-Fi for communication (S354). If communication is possible, the imagemay be transmitted to an external server (S356).

The server determines whether an application of the terminal used by theuser is synchronized (S358). If the application is synchronized, theimage may be displayed on the display 14 of the terminal.

Accordingly, the user may receive information about the times stored inthe refrigerator through the user's terminal.

In the case that the refrigerator is not connected with Wi-Fi to performexternal communication or that the refrigerator is not synchronized withthe application of the terminal, the last image is preferably maintainedon the display 14 of the user's terminal and provided for the user.

Hereinafter, a description will be given of a constituent provided tocapture a photo and acquire the photo at a desired time.

When the door switch senses that the user begins to open the door, thecontroller drives the camera to photograph the interior of therefrigerator. The camera continuously captures a certain number offrames per second. The user's action of opening the door represents theuser's intention to retrieve a desired item from the refrigerator orstoring a new item in the refrigerator. Accordingly by continuouslyphotographing the interior of the refrigerator, change of the conditionof the storage space in the storage compartment of the refrigerator maybe monitored.

The storage compartment is divided by shelves, drawers or baskets toform the storage space. Items are stored in the storage spaces accordingto the shapes of the items or the vessels or content.

The camera may monitor the storage space divided into a shelf region,drawer region and other storage regions by the shelves. Thereby, thecamera allows the user to check the stored condition of the items in therespective region immediately after the user completes the storing orretrieving operation. In the case that two drawers are provided and twoother storage regions are disposed in parallel, information allowing theuser to check the two drawers and the two storage regions may beprovided through the camera.

According to one implementation, the other storage regions are formed atthe bottom of the storage compartment, and a space which is positionedin front of the storage region of the drawer and at least partiallyoverlap the drawers when the drawers open is set as an accommodationspace to be monitored.

When the door switch senses opening of the door, the controller turns onthe door sensor provided to a hinge assembly of the refrigerator tosense movement according to rotation of the door. The door sensor may bekept turned on, or may be turned only when the door switches sensesopening of the door.

When the door is opened by an angle greater than or equal to a certainangle, the controller determines that the door is substantially openedto allow the user to access the storage compartment to store or retrievean item, and determines the storing condition of the shelf region as anobject to be upgraded.

In addition, the drawer sensing unit senses the degree of opening of thedrawer. In the case that the drawer is open, it determines the storingcondition of the drawer region as an object to be upgraded. In the casethat the drawer is not open, it determines the accommodation region asan object to be upgraded.

According to one implementation, the door sensor senses a certain anglethat the door passes when the door is opened or closed and then sendsrelevant information to the controller. Herein, the door sensor may alsosense the direction of rotation of the door, i.e. the closing directionor the opening direction.

According to one implementation, when an ON signal of the door switch issensed, the controller turns on the door sensor. The door sensor in turnsenses the time at which the door passes a certain angle and sends thetime to the controller. Then, the controller determines that the door isopen at this time.

Specifically, the door has rotated the certain angle and is in the openstate.

When the user finishes retrieving or storing an item with the door open,the user closes the door. When the door passes by the door sensor, thedoor sensor sends a signal to the controller. Then, the controllerdetermines, based on this signal, that the user completes the storing orretrieving operation and the door is about to be closed. Specifically,the door is open by the certain angle at this time and rotates to beclosed.

Hereinafter, a process of capturing photos of the interior of therefrigerator during opening or closing of the door and updating thephotos immediately after the user completes storing or retrieving of anitem according to one implementation will be described in detail.

When the door switch is turned on with the door closed, the controllerturns on the camera and the door sensor. This means that the camera isswitched from the standby mode to the drive mode and the door sensor isswitched to a mode in which the door sensor is capable of determiningthe angle of opening of the door.

The camera may continuously photograph the interior of the refrigeratorat a predetermined number of frames per unit time. The captured photo issent to the controller, and the controller temporarily buffers thetransmitted photo as an image including the entire interior of therefrigerator.

In this specification, the term “picture (or photo)” may bedistinguished from the term “image.”

Specifically, the term “picture (or photo)” may represent raw datatransmitted immediately after being captured, and the term “image” mayrepresent data obtained by buffering, correcting or transmitting apicture (or photo), which is raw data, through the controller.

When the door switch is turned on the controller determines that thedoor is open according to a certain angle of opening of the door, thecontroller determines which region the whole image shows as storingitems, according to opening of the drawer.

For example, in the case that the drawer is open, the controllerdetermines that the stored whole image shows items stored in a region(shelf region) over a shelf of the refrigerator and a drawer storageregion (drawer region) in the drawer. On the other hand, when it isdetermined that the drawer is closed, the controller determines that thestored whole image shows items stored in the shelf region of therefrigerator and an internal region of the accommodation space which isformed at the front edge of the drawer and overlaps the drawer when thedrawer is opened.

In a photo of the entirety of the storage compartment captured andtransmitted by the camera, the storage compartment is divided into threedifferent storage spaces. The shelf region is always contained in thephoto, while the drawer region and the accommodation region areselectively contained in the photo according to the opening/closingstate of the drawer. The drawer region and the accommodation region donot coexist in the photo.

Accordingly, one whole photo transmitted from the camera contains atleast two regions. One of the two regions is the shelf region and theother storage region is determined by the opening/closing state of thedrawer.

The present invention is directed to providing the user with informationabout the latest stored condition of items by capturing two differentstorage spaces including at least the inner space of the drawer with acamera. Therefore, photographing only the drawer region and the shelfregion and updating the corresponding photo may be one implementation,and photographing only the drawer region and the accommodation space andupdating the corresponding photo may be another implementation.

Meanwhile, dividing a whole photo of the storage compartmentconstructing one independent insulated space in the refrigerator intothe storage regions in the storage compartments may be determined thenumber of the doors that opens and closes the storage compartment. Forexample, in the case of a refrigerator having a storage compartment thatis opened and closed by one door, a captured photo can be divided intotwo or three storage regions. In the case that the storage compartmentis opened and closed by left and right doors, a captured photo may bedivided into four or six storage regions.

In this case, each of the drawer region and the accommodation region maybe vertically divided such each of them are shown in two divided areasin a whole photo, and the shelf region may be shown in one area. In thiscase, one whole photo may include five divided areas.

The storage spaces of the shelf region facing the left and right doorsare connected to each other. In the shelf region, goods, food items orvegetables may be laterally arranged from the left side to the rightside of the shelf region. In this case, items may be stored in orretrieved from the shelf region by access opening only one of the leftand right doors. Accordingly, the whole shelf region in a whole capturedimage needs to be replaced with latest data when at least one of the twodoors is open. In this context, the whole shelf region may be updatedwhen at least one of the left and right doors is open.

More specifically, when it is determined that the drawer is opened withthe door open, the controller separates the shelf region and the drawerregion from a transmitted photo and temporarily buffers the same. Inaddition, when the user is about to close the drawer after completingstoring or retrieving an item in or from the drawer region and/or theshelf region, that is, when the drawer which has been stopped openbegins to move, a necessary part in the drawer region in a photo mostrecently captured among the temporarily buffered photos is captured orcut and stored in the storage unit as an image showing the lateststorage condition of the drawer region. At the same time, in a photo ofthe interior of the refrigerator displayed on the display, the capturedimage of the drawer region is upgraded by replacing the image with thelatest photo. In addition, the controller transmits the captured imageof the drawer region to a server system connected to the refrigeratorover a network.

Meanwhile, when access opening of the drawer is not sensed with the dooropen, and thus it is determined that the drawer is closed, thecontroller determines that the whole image of the transmitted photocontains the shelf region and the accommodation region. Accordingly, theshelf region and the accommodation region are separated from the wholeimage and temporarily buffered.

When the user closes the drawer after completing storing or retrievingan item in or from the drawer region and/or the shelf region, the doorsensor senses the door when the door passes a predetermined angle andtransmits a signal to the controller. The controller in turn determines,based on the signal transmitted from the door sensor, that the door isabout to be closed (which means that photographing is about to beobstructed by an obstacle such as the door), and then captures anecessary part in the accommodation region in a photo most recentlycaptured among the temporarily buffered photos and stores the same inthe memory as an image showing the latest storage condition of thedrawer region. At the same time, in the photo of the interior of therefrigerator displayed on the display, the captured image of theaccommodation region is upgraded by replacing the image.

Herein, upgrading may mean replacing an existing image with the latestimage. The controller transmits the captured image of the drawer regionto the server system connected to the refrigerator over a network.

When the drawer is open or closed, the latest image of the shelf regionis captured among the images buffered at the time when it is determinedthat the door is about to be closed, and then stored in the memory asthe image showing the latest stored condition of the items in the shelfregion.

Replacing the image displayed on the display of the refrigerator withthe captured image (one or all of the latest image of the shelf regionand the latest image of the drawer region or the accommodation region)or transmitting the captured image to the server system or a mobiledevice of the user connected to the refrigerator through wirelesscommunication may be conducted at the same time the captured image isstored in the memory of the refrigerator. Such operations may beconducted all together when there is a request from the user or may beconducted for individual regions according to a request signal from theuser.

One photo transmitted from the camera may be buffered as one image bythe controller. When it is determined that the image has been capturedwith the door open, whether the image contains the drawer region or theaccommodation region may be determined depending upon whether the draweris open or closed. According to the determination, a corresponding imageis extracted and buffered in a temporary buffer.

When it is determined that the one image has been captured at the timewhen the drawer begins to be closed, the latest image is stored in thememory among the images which are being temporarily buffered. A newstoring or retrieving action is completed by the user, the existingimage is replaced with the latest image. In addition, when it isdetermined that the door is about to be closed, the latest image isstored in the memory among the images of the accommodation region whichare being temporarily buffered.

Meanwhile, a portion of the latest image showing the shelf region may beextracted from the latest image before the door is about to be closedamong the images of the accommodation region which are being temporarilybuffered, and then stored in the memory. A separate temporary buffer forthe shelf region may be provided and only the portion showing the shelfregion may be extracted and buffered in the separate buffer.

Hereinafter, descriptions will be given of transmitting an image of theinterior of the refrigerator to a personal mobile terminal or a serverof a server provider with the illustrated various methods to utilize theimage according to various examples, with reference to FIGS. 52 to 55.

FIG. 52 is a ladder diagram illustrating a method of operating therefrigerator according to another example.

Referring to FIG. 52, when it is determined that a final photo of aspecific region in the refrigerator has been acquired through the stepsas described with reference to FIG. 34 (S1101), the controller 100transmits a final image obtained through correction to the terminal 2 asan event (S1103). The terminal 2 may include a smartphone, a persondigital assistant (PDA), and a tablet PC. However, examples are notlimited thereto. The controller 100 may transmit not only the finalimage of the interior of the refrigerator but also various kinds ofinformation related to the final image to the terminal 2.

At this time, the controller 100 transmit the final image to theterminal 2 upon acquiring the final image through the final photo, ormay store the acquired final image in the storage unit 18 and thentransmit a stored image to the terminal 2 on a predetermined cycle as anevent. However, implementations of the present invention are not limitedthereto.

Subsequently, the terminal 2 displays the final image received accordingto the user's selection (S1105). For example, the terminal 2 execute anapplication capable of receiving and displaying the image of theinterior of the refrigerator according to the user's selection.Otherwise, the terminal 2 may perform a multimedia service to displaythe received image of the interior of the refrigerator.

According to this example, the user can check the image of the interiorof the refrigerator using a mobile terminal without looking into therefrigerator. Therefore, the user may remotely understand the currentcondition of the interior of the refrigerator.

FIG. 53 is a ladder diagram illustrating a method of operating therefrigerator according to another example.

Referring to FIG. 53, when it is determined that a final photo of aspecific region in the refrigerator has been acquired as described above(S1101), the controller 100 produces images by dividing the final photoaccording to the respective regions, and then controls the storage unit18 to store an acquired final image (S1303).

Subsequently, when the terminal 2 receives a user input for displayingthe final image of the specific region of the refrigerator (S1305), theterminal 2 transmits a request for the final image to the refrigerator(S1307).

In response to the request for the final image, the controller 100controls communication unit 270 to transmit an image of the interior ofthe refrigerator most recently captured and stored among the imagesstored in the storage unit 18 to the terminal 2 (S1309).

According to this example, communication between the mobile terminal andthe refrigerator is conducted only when the user desires. Thereby, thecommunication may become more efficient and economical. Hereinafter, adescription will be only given of a case of transmitting an image inresponse to the request. Transmitting of an image includes transmittingan image as an event as described above with reference to FIG. 35.However, implementations are not limited thereto.

FIG. 54 is a ladder diagram illustrating a method of operating therefrigerator according to another example.

Referring to FIG. 54, when it is determined that a final photo of aspecific region in the refrigerator has been acquired as described above(S1501), the controller 100 controls the storage unit 18 such that afinal image acquired from the final photo is stored in the storage unit18 (S1503).

Subsequently, when a server of a server provider such as a market server3 receives a user input for displaying a current image of the interiorof the refrigerator (S1505), the market server 3 transmits a request forthe final image to the refrigerator (S1507). At this time, the marketserver 3 may transmit a request for the final image on a predeterminedcycle without the user input.

Then, the controller 100 performs a control operation in response to theimage request such that the final image of the interior of therefrigerator most recently captured and stored among the images storedin the storage unit 18 is transmitted to the market server 3 (S1509).

Upon receiving the final image of the interior of the refrigerator, themarket server 3 analyzes the final image (S1511). By analyzing the finalimage, the market server 3 may identify items currently stored in therefrigerator and items currently not present in the refrigerator andanalyze the items previously stored but currently not present in therefrigerator to extract information necessary for provision of aservice.

The market server 3 transmits a list of inadequate items extracted byanalyzing the final image to the refrigerator (S1513).

FIG. 55 is a ladder diagram illustrating a method of operating therefrigerator according to another example.

Referring to FIG. 55, when it is determined that a final photo of aspecific region in the refrigerator has been acquired as described above(S1701), the controller 100 controls the storage unit 18 such that afinal image acquired by correcting the final photo is stored in thestorage unit 18 (S1703).

Subsequently, when a server of a server provider such as a broadcastingstation server 4 receives a user input for displaying the final image ofthe interior of the refrigerator (S1705), the broadcasting stationserver 4 transmits a request for the final image to the refrigerator(S1707). At this time, the broadcasting station server 4 may transmit arequest for the final image on a predetermined cycle without the userinput.

Then, the controller 100 performs a control operation in response to therequest for the final image such that the final image of the interior ofthe refrigerator most recently captured and stored among the imagesstored in the storage unit 18 is transmitted to the broadcasting stationserver 4 (S1709).

Upon receiving the final image of the interior of the refrigerator, thebroadcasting station server 4 analyzes the final image (S1711). Byanalyzing the final image, the broadcasting station server 4 mayidentify items currently stored in the refrigerator and to extractinformation necessary for provision of a service, for example, byanalyzing dishes which may be prepared using the currently stored items.

Through the analysis of the final image, the broadcasting station server4 transmits, to the refrigerator, information about a recipe of a dishwhich may be prepared using the currently stored items (S1713).

The controller 100 displays the information about the recipe of a dishreceived from the broadcasting station server 4 on the display 14installed at the refrigerator.

According to this example, an involved provided may accurately identifythe condition of the interior of the refrigerator positioned at home ora business and provide a refrigerator-related proper service for thehome or business.

FIGS. 56A and 56B are views illustrating operation of a heater of thecamera, and FIGS. 57A and 57B are views illustrating a result of anexperiment on dew condensation occurring on a transparent window of thecamera according to temperatures. Hereinafter, a description will begiven with reference to FIGS. 56A and 56B. FIGS. 56A and 56B depicts theresult based on the heater of the type described with reference to FIGS.14 to 17.

FIG. 56A is a graph depicting discontinuously driving the heater 84 withpower discontinuously supplied thereto, and FIG. 56B is a graphdepicting continuously driving the heater 84 with power suppliedcontinuously thereto.

In FIG. 56A, the temperature of the transparent window 80 is notcontrolled in normal times, but remains balanced with the temperature ofthe storage compartment 22. At the time when photographing therefrigerator with a camera is needed, high power is instantaneouslyapplied. Particularly, the heater 84 may be driven to remove dew fromthe transparent window 80 immediately before photographing is performed.

When the door 20 is opened, the transparent window 80 may contactmoisture contained in external warm air introduced through the door 20.At this time, the transparent window 80 is at a low temperature relativeto the external air, and accordingly moisture contained in the externalair may be condensed on the transparent window 80, thereby causing dewcondensation.

In the case that the heater 84 is instantaneously driven to remove dewformed by being condensed, time may be taken until the dew on thetransparent window 80 is eliminated. Accordingly, in the case that thetime to capture a photo is earlier than the time at which the dew isremoved, dew may be present on the transparent window 80 at the timewhen photographing is performed.

On the other hand, in FIG. 56B, the heater 84 is continuously driven tosupply heat to the transparent window 80. Since the transparent window80 is constantly maintained at a temperature greater than thetemperature at which dew condensation occurs, dew condensation does notoccur on the transparent window 80. Accordingly, even if the door 20 isopened and thus the transparent window 80 contacts the external air, themoisture contained in the external air is not condensed on thetransparent window 80. Therefore, dew is prevented from being present onthe transparent window 80 at the time when the camera 70 instantaneouslycaptures a photo.

In driving the heater 84, the dew condensation should be prevented fromaffecting the photo quality. Also, rapid increase of power consumed bythe heater 84 should be prevented. In addition, heat supplied from theheater 84 should be preferably prevented from affecting the internaltemperature of the refrigerator.

It has been found that the amount of time for which the heat is turnedoff needs to be increased to minimize the power consumption of theheater and prevent the heat from affecting the interior of therefrigerator. However, considering that increasing the time for whichthe heater is turned off may result in dew condensation, experimentshave been conducted by setting a target temperature of the transparentwindow of the camera under the condition that the temperature of the airoutside the refrigerator is 32° C., and relative humidity is 85% (i.e.,under that condition that the temperature of the dew point is betweenabout 29° C. and about 30° C.) (see FIGS. 57A and 57B). In FIG. 57A, they-axis represents the amount of condensed dew. In FIG. 57B, the y-axisrepresents the amount of dew actually condensed at the correspondingtemperature, which is expressed in percentage relative to the currentamount of vapor.

The On/Off cycle of the heater derived from repeated experiments maydeteriorate the quality of photos captured by the camera, under aspecial temperature condition.

Even if a proper cycle to turn on/off the heater is found, the patternof use of the is refrigerator by the user may unexpectedly vary for thetime for which the heater is turned off.

For example, in the case that the user leaves the door open for a longtime, a large amount dew may be formed on the surface of a cover glass(the transparent window or other types of windows placed at the frontedge of the camera lens) of the camera. In this case, a lot of time istaken to cause evaporation even if the cover glass is heated by drivingthe heater. Accordingly, the quality of a photo captured at this timemay be low. In addition, evaporation the dew formed on the cover glasstakes a lot of power as well as a large amount of time.

Moreover, in the case that the storage compartment is the fresh foodstorage compartment, the temperature of the interior of the storagecompartment is usually maintained between 0° C. and 7° C., and theinternal volume of the storage compartment is greater than the surfacearea of the cover glass. A large portion of the heat supplied from theheater may be exposed to the cooled air in the storage compartment.Accordingly, a large amount of heat needs to be applied in a short timeto evaporate the dew formed on the cover glass. To intensively supplyheat in a short time, a heat having a large capacity may need to beused. However, in this case, the power that is instantaneously used maybe greatly increased, and thus various problems may be caused whensupply of power is not smooth.

Accordingly, the amount of dew condensed on the cover glass ispreferably controlled consistently not to become greater than or equalto a certain amount from the earliest stage. In some implementations,power is applied to the heater such that the heater is constantly drivenwithout an On/Off cycle.

In applying a constant power to the heater, the heater of the camera maybe configured to receive power upon application of the power.Alternatively, power may be controlled through the controller of therefrigerator such that constant power is applied to the heater while thestorage compartment is normally operated.

Next, the temperature of the heated cover glass will be described. Whenthe qualities of sample photos obtained at different temperature rangeswere evaluated, it was found that any significant problem with thequality of the sample photos does not occur until the temperature of thecentral portion of the cover glass reaches half the dew point. Having noproblem with the photo quality may mean that the kinds and quantities ofthe items stored in the storage compartment can be identified.

Particularly, in FIG. 57B, a photo sample obtained at a third point,which indicates 13.5° C. has an allowable quality. Power may be suppliedto the heater such that the temperature of the heated cover glass, whichmay be maintained to be greater than or equal to the dew point, ismaintained between about 13° C. and about 30° C.

In the result of the experimentation, it has been found that the heatingtemperature of the cover glass varies with the outer size of the coverglass, but the thickness of the cover glass little varies, under thecondition that the same power is supplied. Since the photo quality wassatisfactory at a temperature which is half the dew point when the sizeof the cover glass is constant, this temperature may be set to be thelower limit, and the upper limit may be preferably set in the aspect ofconsumed power (or applied power) of the heater.

In heating the cover glass, the power applied at a temperature close tothe dew point (29.2° C.) was 0.45 W, and the power applied at atemperature (17.7° C.) about 0.58 times the dew point was 0.27 W. Whenthe heating temperature is about 0.5 to 0.7 times the dew point, theconsumed power is also about 0.5 times the power consumed at the dewpoint. Therefore, power consumption has been effectively reduced in thisrange of temperature.

Under a condition of external air, which is usually changeable, therange of temperature that is about half the dew point may be determinedto be between about 12° C. and about 17° C. Accordingly, electricalcurrent allowing the central portion of the transparent window to bemaintained at a temperature between about 12° C. and about 17° C. ispreferably applied to the heater.

That is, in this example, power consumed by the heater may be reduced bymaintaining the cover glass not at the dew point but at a temperaturelower than the dew point

FIG. 58 is a cross-sectional view illustrating a transparent window.Hereinafter, a description will be given with reference to FIG. 58.

Hydrophilic coating may be applied to the transparent window 80. Thetransparent window 80 may be the cover 90 as described above.

Even if dew is instantaneously formed on the transparent window 80, thecontact angle of water may be formed to be within 15 degrees by thehydrophilic coating, as shown in FIG. 58. When the hydrophilic coatingis applied to a surface, the surface tension of water on the surface isweakened. Accordingly, water may be spread wide on the surface of thetransparent window 80. Thereby, the camera may be fixed to the surfaceof the ceiling of the storage compartment such that the angle betweenthe transparent window and the surface of the ceiling of the storagecompartment is between about 10 degrees and about 20 degrees.

Therefore, it may be possible to minimize distortion of a captured photoresulting from water formed on the transparent window 80. In addition,once the hydrophilic coating is applied, additional control such assupplying electricity is not needed thereafter. Accordingly, it may havean advantage in the aspect of energy efficiency.

FIGS. 59A to 60B are views schematically illustrating installation ofthe camera at the inner case. FIGS. 59A and 60A show views of thestorage compartment seen from one lateral side, and FIGS. 59B and 60Bshow views of the inner case seen from a position below the inner case.Hereinafter, a description will be given with reference to FIGS. 59A to60B. In the examples of FIGS. 59A to 60B, a path along which theexternal air reaches the cover 90 may be arranged in a detouring mannerto minimize condensation occurring on the cover 90.

The camera 70 may be installed at a protrusion protruding downward fromthe ceiling of the inner case 12. Herein, the protrusion 500 may be aportion of the inner case 12 protruding downward relative to the otherportion of the inner case 12. Such protrusion 500 may be formed by thecamera housing as described above.

As shown in FIGS. 59A and 59B, the camera 70 is installed inside theprotrusion 500 such that it faces an inner sidewall of the storagecompartment 22. The camera 70 may capture a photo of the interior of thestorage compartment 22 through the cover 90. Accordingly, the cover 90is disposed at a position on the protrusion 500 facing the rear wall ofthe storage compartment 22.

When the cover 90 contacts moisture contained in the external air, themoisture is cooled and thus dew is formed on the cover 90.

However, the cover 90 is disposed facing the rear wall of the storagecompartment 22, and thus the external may take a long path to reach thecover 90.

That is, for the external air passing by the door 20 to pass through thestorage compartment 22 and contact the cover 90, the external air needsto takes the path as shown in FIGS. 59B and 60B. At this time, some ofthe moisture contained the external air may be condensed on a portion ofthe storage compartment 22 as the external air contacts the cooled airin the storage compartment 22. Accordingly, the external air may reachthe cover 90 while the amount of the moisture therein is graduallyreduced.

That is, when the door 20 is opened, the interior of the storagecompartment 22 may be maintained at a lower temperature than theexterior of the storage compartment 22, and the temperature thereof maynot be immediately changed to the temperature of the external air.Accordingly, heat exchange with the external air occurs as the externalair enters the storage compartment 22.

In other words, in the case that the cover 90 is disposed at a portionwhere it is relatively difficult for the cover 90 to contact theexternal air, the amount of dew formed on the cover 90 may be reduced,and dew may be formed later.

First, as the external air enters the refrigerator, it contacts thefront surface 501 of the protrusion 500. At this time, the external aircontacting the front surface 501 is introduced further into therefrigerator along the path curve to the left and right sides of theprotrusion 500. Such external air may reach the cover 90 whilecontacting the side surface of the protrusion 500. Accordingly, a largeamount of moisture may be condensed before the external air reaches thecover 90.

In addition, the cover 90 is mounted at a place surrounded by the frontsurface 501, side surfaces 502 and the lower surface 503 such that thecover 90 is inclined. Thereby, the cover 90 is not arrangedperpendicular to the path of the external air. Accordingly, the externalair may be prevented from being rapidly condensed when it reaches thecover 90.

Particularly, the width of the protrusion 500, specifically, the lateralwidth of the front surface 501 may be greater than the width of thecamera 70.

The example illustrated in FIGS. 60A and 60B, in which the camera ismounted facing vertically downward, is very similar to the example ofFIGS. 59A and 59B. Accordingly, the example of FIGS. 60A and 60B mayhave the same effect as that of the example of FIGS. 59A and 59B.

In addition, the protrusion 500 may include a rear surface 504.Preferably, the vertical length of the front surface 501 is greater thanthe vertical length of the rear surface 504. That is, the front surface501 may protrude farther downward of the inner case 12 than the rearsurface 504.

Accordingly, the path along which the external air reaches the cover 90is elongated in the horizontal direction by the width of the protrusion500 and elongated in the vertical direction by the protruding height ofthe protrusion 500.

Prevention of dew condensation on the cover 90 through such structure ofthe protrusion 500 or the mounting structure of the camera is closelyrelated the position at which the camera is installed. That is, thecamera may be installed to be inclined rearward at a predetermined anglewithin a certain range with respect to a vertical line extendingvertically downward from the ceiling of the storage compartment.Accordingly, the introduction path of the external air may beeffectively detoured while the entire regions in the storage compartmentare effectively photographed.

In conventional art, the camera is mounted to a sidewall of the storagecompartment of the cabinet, an outer upper end of the cabinet, the dooror the upper portion of the door.

According to the conventional art, in the case that the camera ismounted to a sidewall of the storage compartment, a recessed portioncapable of accommodating the camera is provided to the sidewall, and thecamera is mounted to the recessed portion. That is, the camera does notprotrude into the storage compartment. This structure may not cause userinconvenience when a shelf of the storage compartment or an item isretrieved from the storage compartment.

In contrast, in some implementations, the camera is installed at theceiling of the inner case 12. Thereby, the user is not obstructed by thecamera when the user uses the storage compartment 22. This is becausethe camera is not positioned in the path of movement of the shelves andthus collision with the camera is prevented when the user desires toretrieve an item from the storage compartment.

Further, in some implementations, the cover 90 of the camera 70 isdisposed to face in the rearward direction. Thereby, the path alongwhich the external air reaches the cover 90 may be elongated.

In FIG. 59A, the cover 90 is disposed to face in the rearward direction,and the protrusion 500 protrudes downward. Accordingly, for the externalair to reach the cover 90, the external air needs to follow athree-dimensional path of movement and thus the path of movement of theexternal air is elongated.

In FIG. 60A, the cover 90 is disposed to face downward, and theprotrusion 500 is provided with a front surface 504 disposed in parallelwith the access opening and side surfaces 508 disposed perpendicular tothe front surface 504. Herein, the front surface 504 and the sidesurfaces 508 are installed to protrude further downward such that thepath along which the external air reaches the transparent window 80 maybe elongated.

That is, the high-temperature air introduced from the outside of theaccess opening of the storage compartment does not straightly reach andcontact the cover 90, but reaches the protrusion 500 by detouringleftward, rightward and downward along the protrusion 500. Accordingly,as the external air exchanges heat with the cooled air in therefrigerator, it is cooled and thus has a reduced amount of saturatedwater vapor. Thereby, the water vapor contained in the external air iscondensed and thus mixed with the cooled air in the refrigerator. Whenthe external air finally reaches the cover 90, it may be sufficientlyevaporated by the heater which heats the cover 90 since the external aircontains a reduced amount of water vapor.

Accordingly, the power to be provided to the heater 84 may decrease, andthus the overall power consumption of the refrigerator may be improved.

In the examples illustrated in FIGS. 59A to 60B, the refrigerator mayinclude a cabinet provided therein with a storage compartment configuredwith an access opening formed in the front surface of the storagecompartment and formed by an upper wall, a lower wall, both sidewallsand a rear wall which are formed of an insulating material, the storagecompartment including a shelf region divided by a plurality of shelvesand a drawer region having at least one drawer, at least one doorclosely contacting the front surface of the storage compartment to openand close the storage compartment, and a camera device installed at theupper wall of the storage compartment between the access opening of thestorage compartment and a front edge of the shelves installed in therefrigerator.

The camera device may include a camera module part configured to receivea camera lens and electrical components needed to drive a camera andprovided with a transparent window spaced a certain distance from thecamera lens, and a camera housing part configured to seat and fix thecamera module part at a determined position in the camera housing part.

The camera housing part may include a fixing surface adjoining the upperwall of the storage compartment, a front surface formed to face theaccess opening of the storage compartment, both side surfaces connectingthe front surface to the rear surface, and a top surface provided withan opening allowing a camera window of the camera module to be exposedtherethrough. The front surface of the camera housing part is formed ata higher position than the opening formed in the top surface of thecamera housing part.

The top surface of the camera housing may be inclined at a certain anglewith respect to the surface of the upper wall of the storage compartmentto face away from the opening of the storage compartment.

The camera housing part may further include a rear surface at a positionopposing the front surface and facing the rear wall of the storagecompartment. The height of the front surface may be greater than theheight of the rear surface.

The opening in the top surface of the camera housing part may be formedat a lower position than the side surface of the camera hosing part.

The refrigerator may further includes a housing installation part havinga recessed portion corresponding to a recessed space recessed inward ofthe upper wall of the storage compartment.

The housing installation part is installed at the upper wall of thestorage compartment when the refrigerator is filled with an insulatingmaterial, and the camera housing part may be fixed to the housinginstallation part.

The camera module part fixed to the camera housing part may beaccommodated in the recessed portion formed at the housing installationpart.

The camera module part accommodated in the recessed portion of thehousing installation part may be fixed to the camera housing part suchthat a gap is formed between the camera module part and the innersurface of the housing installation part.

The front surface and side surfaces of the camera housing part may beformed at a higher position than the opening in the top surface of thecamera housing part.

A portion of the top surface of the camera housing part close to theopening of the storage compartment may be farther apart from the surfaceof the upper wall of the storage compartment than the other portion ofthe top surface close to the rear wall of the storage compartment.

The camera module may include a front portion provided with the camerawindow and a rear portion opposing the front portion. At least one ofthe front portion and the rear portion may be provided with at least oneflat surface.

A receiving portion to receive the camera module part may be formed atan inner side of the camera housing part. The receiving portion mayinclude a seating part inclined at a certain angle with respect to thesurface of the upper wall of the storage compartment to allow the flatsurface formed at the camera module part to be placed thereon.

The recessed portion of the housing installation part may include aseating part inclined at a certain angle with respect to the surface ofthe upper wall of the storage compartment to allow the flat surfaceformed at the camera module part to be placed thereon.

A flat surface may be formed around the camera window of the frontportion of the camera module part. The flat surface may be formedapproximately at the same level as the central portion of the cameralens.

FIG. 61 is a view illustrating a refrigerator according to anotherimplementation. Hereinafter, a description will be given with referenceto FIG. 61.

The refrigerator shown in FIG. 61 is a side-by-side refrigerator havingdifferent storage compartments laterally disposed. That is, the leftstorage compartment may be a freezer compartment, and the right storagecompartment may be a fresh food storage compartment.

The camera 70 may be installed at the ceiling of the storage compartment22 such that the camera 70 faces the lower portion of the storagecompartment 22.

In addition, a drawer 50 configured to store food and adapted bewithdrawn and introduced may be installed in the storage compartment 22.

Other details are the same as those described in the previous examples,and thus they will be described.

FIG. 62 is a view illustrating a screen provided for the user from therefrigerator of FIG. 61. Hereinafter, a description will be given withreference to FIG. 62.

An image of the items stored in the drawer 50 and 51 and an image of theitems stored at the upper side of the shelf 40 may be provided for theuser. The refrigerator may be provided with one drawer 50. Accordingly,an image of one drawer may be provided in the example of FIG. 62.Otherwise, two drawers 50 and 51 arranged in a vertical direction may beprovided as shown in FIG. 61. In this case, images of the interior ofthe respective drawers may be provided.

In the example illustrated in FIG. 2, the drawers are laterallyarranged. In contrast, in FIG. 62, the drawers may be verticallyarranged. Accordingly, two drawer regions may be provided, and the lowerdrawer 50 may protrude farther forward than the upper drawer 51. In thiscase, each of the drawers 50 and 51 may be provided with a marker.

Thereby, information about food items stored in a plurality of shelfregions, a plurality of drawer regions may be easily understood throughone camera.

The method of upgrading an image provided on the user screen isidentical to that for the refrigerator as previously described, and thusa description thereof will not be given.

FIG. 63 is a view illustrating a method of adjusting a picture capturedwith a camera of the refrigerator of FIG. 61. Hereinafter, a descriptionwill be given with reference to FIG. 63.

Generally, only one drawer is provided or two drawers are verticallydisposed. Accordingly, in the refrigerator of a type shown in FIG. 61,one adjustment line 15 may be provided. Two horizontal adjustment linesmay be provided to divide two drawers.

Additionally, two vertical adjustment lines may be disposed in paralleland the positions thereof may be adjusted. The horizontal adjustmentline may only be used to select a desired image without using thevertical adjustment line, unlike the example of FIG. 63.

Limit lines 16 indicating the limit range of movement of the adjustmentline 15 may be provided for the user. In this case, the limit lines 16may be disposed in parallel and space a predetermined distance from eachother.

Correction of an error through the adjustment line 15 may be identicalor similar to the correction in a previous example.

Various examples have been described in the best mode for carrying outthe invention.

INDUSTRIAL APPLICABILITY

The present invention provides a refrigerator allowing the user obtaininformation about food stored in the refrigerator without opening thedoor of the refrigerator. Thereby, leakage of cooled air from thestorage compartment may be prevented. Accordingly, unnecessary loss ofcooled air may be prevented and the energy efficiency of therefrigerator may be improved.

In addition, the latest information about the food stored in therefrigerator may be provided for the user. Therefore, reliability of theinformation about the stored food provided for the user may be enhanced.

Further, information about food items stored in various positions may beprovided by a single camera. Accordingly, a structure allowinginstallation of only one camera is added, and thus designing of therefrigerator may be facilitated. Particularly, costs incurred by use ofa camera may be reduced.

Dew condensation on a camera installed in the refrigerator may beprevented. Accordingly, an image captured by the camera may be stablyprovided for the user.

The user may remotely figure out the current internal situation of therefrigerator and receive necessary information about food from anoutside provider.

The refrigerator may provide a screen of a photo of the interior of adrawer captured by a camera which is similar to what the user actuallysees when the user uses the refrigerator.

The refrigerator may provide the user with planar images of spatiallyoverlapping and invisible locations on one screen.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in the present inventionwithout departing from the spirit or scope of the invention. Thus, it isintended that the present invention cover the modifications andvariations of this invention provided they come within the scope of theappended claims and their equivalents.

A number of methods, techniques, systems, and apparatuses have beendescribed. Nevertheless, various modifications may be made withoutdeparting from the scope of this disclosure.

The methods, techniques, systems, and apparatuses described herein maybe implemented in digital electronic circuitry or computer hardware, forexample, by executing instructions stored in tangible computer-readablestorage media.

Apparatuses implementing these techniques may include appropriate inputand output devices, a computer processor, and/or tangiblecomputer-readable storage media storing instructions for execution by aprocessor.

A process implementing techniques disclosed herein may be performed by aprocessor executing instructions stored on a tangible computer-readablestorage medium for performing desired functions by operating on inputdata and generating appropriate output. Suitable processors include, byway of example, both general and special purpose microprocessors.Suitable computer-readable storage devices for storing executableinstructions include all forms of non-volatile memory, including, by wayof example, semiconductor memory devices, such as Erasable ProgrammableRead-Only Memory (EPROM), Electrically Erasable Programmable Read-OnlyMemory (EEPROM), and flash memory devices; magnetic disks such as fixed,floppy, and removable disks; other magnetic media including tape; andoptical media such as Compact Discs (CDs) or Digital Video Disks (DVDs).Any of the foregoing may be supplemented by, or incorporated in,specially designed application-specific integrated circuits (ASICs).

Although the operations of the disclosed techniques may be describedherein as being performed in a certain order and/or in certaincombinations, in some implementations, individual operations may berearranged in a different order, combined with other operationsdescribed herein, and/or eliminated, and desired results still may beachieved. Similarly, components in the disclosed systems may be combinedin a different manner and/or replaced or supplemented by othercomponents and desired results still may be achieved.

The foregoing implementations and advantages are merely examples and arenot to be considered as limiting the present disclosure. The presentteachings can be readily applied to other types of apparatuses. Thisdescription is intended to be illustrative, and not to limit the scopeof the claims. Many alternatives, modifications, and variations will beapparent. The features, structures, methods, and other characteristicsof the implementations described herein may be combined in various waysto obtain additional and/or alternative implementations.

1. (canceled)
 2. A refrigerator comprising: a cabinet in which a storagecompartment is defined by a fixed insulation wall, the storagecompartment configured to store food and having an access opening; adoor rotatably provided at the cabinet and configured to open and closethe access opening of the storage compartment; a camera configured totake a photo that captures a plurality of regions of an interior of thestorage compartment; a display unit; and at least one processorconfigured to control the display unit to display, simultaneously in ascreen, a plurality of images that represent the plurality of regions,wherein the plurality of images displayed on the screen of the displayunit are arranged in a row and separated from each other, each imagehaving the same width.
 3. The refrigerator according to claim 2, whereinthe at least one processor is further configured to control the displayunit to display the plurality of images by controlling the displaying ofeach of the plurality of images independently from each other.
 4. Therefrigerator according to claim 3, wherein the at least one processor isconfigured to control the displaying of each of the plurality of imagesindependently from each other by: updating the displaying of a first setof images among the plurality of images with new images; and maintainingthe displaying of a second set of images among the plurality of images.5. The refrigerator according to claim 3, wherein the at least oneprocessor is further configured to control the display unit to displaythe plurality of images by: controlling the display unit to display theplurality of images to be spaced apart from each other by apredetermined distance.
 6. The refrigerator according to claim 2,wherein the at least one processor is further configured to control thedisplay unit to display the plurality of images by: controlling thedisplay unit to display the plurality of images such that the pluralityof regions are displayed separately from each other.
 7. The refrigeratoraccording to claim 2, wherein the at least one processor is furtherconfigured to control the display unit to display the plurality ofimages by: controlling the display unit to display the plurality ofimages corresponding to a vertical arrangement of the plurality ofregions in the storage compartment.
 8. The refrigerator according toclaim 2, wherein the at least one processor is further configured tocontrol the display unit to display the plurality of images by:controlling the display unit to display the plurality of imagescorresponding to a horizontal arrangement of the plurality of regions inthe storage compartment.
 9. The refrigerator according to claim 2,wherein the at least one processor is further configured to control thedisplay unit to display the plurality of images by: controlling thedisplay unit to display a frame surrounding the plurality of images. 10.The refrigerator according to claim 9, wherein the plurality of regionsis displayed within the frame in the screen of the display unit.
 11. Therefrigerator according to claim 9, wherein the plurality of images havea same vertical width.
 12. The refrigerator according to claim 2,wherein the at least one processor is further configured to control thedisplay unit to display the plurality of images by: controlling thedisplay unit to display at least two regions, among the plurality ofregions, that overlap each other in the storage compartment.
 13. Therefrigerator according to claim 2, further comprising a communicationdevice configured to send at least a portion of the plurality of imagesto an external terminal.
 14. The refrigerator according to claim 2,wherein the at least one processor is further configured to divide thephoto taken by the camera into a first image capturing a first region ofthe plurality of regions, and a second image capturing a second regionof the plurality of regions.
 15. The refrigerator according to claim 14,wherein the first region is a region that stores food in an externalspace of a drawer, and the second region is a region that stores food inan internal space of the drawer.
 16. The refrigerator according to claim2, wherein the at least one processor is further configured to dividethe photo taken by the camera into a first image capturing a firstregion of the plurality of regions, and a third image capturing a thirdregion of the plurality of regions.
 17. The refrigerator according toclaim 16, wherein the first region is a region that stores food in anexternal space of a drawer, and the third region is a region under thedrawer and vertically overlaps the drawer in a state in which the draweris withdrawn.
 18. The refrigerator according to claim 15, furthercomprising a communication device configured to send one of the firstimage or the second image to an external terminal.
 19. The refrigeratoraccording to claim 17, further comprising a communication deviceconfigured to send one of the first image or the third image to anexternal terminal.
 20. The refrigerator according to claim 15, whereinthe camera is further configured to take the photo based on a closing ofthe drawer.
 21. The refrigerator according to claim 9, wherein the atleast one processor is further configured to individually replace theplurality of images within the frame with updated images of theplurality of regions.